Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly 14× faster and using less than one-eighth of the GPU memory.
Setting. A frozen pre-trained encoder $\varphi$ maps images to features; only a linear probe is trained downstream. The distillation goal is a tiny synthetic set $\mathcal{S}_{\text{syn}}$ whose induced probe generalizes on real data.
The learnable synthetic images are passed through the frozen backbone to obtain a feature matrix.
Linear probing on frozen features is a convex quadratic problem — it does not need to be unrolled.
The minimizer is available analytically — no inner loop, and no infinite-width (NTK) approximation.
In the low-IPC regime the number of synthetic samples $N$ is far below the feature dimension $d$. The push-through identity turns a $d \times d$ solve into a much cheaper $N \times N$ solve — this is where most of the speed and memory savings come from.
Instead of an MSE fit to one-hot targets, the induced probe is evaluated discriminatively on real features with a temperature-scaled softmax cross-entropy. Each column $\mathbf{w}^{\star}_j$ behaves as a learned class anchor in feature space.
Why it is cheap. The inner problem is solved once, exactly, as an $N \times N$ linear system — no unrolled optimizer states to store and no backward pass through a trajectory. This is what buys the 14× speedup and the 8× memory reduction reported below.
Per-iteration cost on ImageNet-100 with DINOv2 at IPC = 1. CLP-DD runs at the cost of DSA-free LGM while reaching higher accuracy than LGM with DSA on this backbone setting.
| Method | DINOv2 | MoCo-v3 | CLIP | EVA-02 | Average |
|---|---|---|---|---|---|
| Random | 74.8±2.6 | 61.3±2.6 | 57.6±1.5 | 64.4±2.5 | 64.5±2.5 |
| Centroid | 86.9±0.3 | 77.7±0.1 | 77.1±0.1 | 80.9±0.2 | 80.6±0.2 |
| Neighbor | 86.0±0.2 | 77.1±0.2 | 67.8±0.2 | 78.8±0.1 | 77.4±0.2 |
| LGM (no DSA) | 82.6±0.2 | 59.5±0.5 | 58.4±0.3 | 73.9±0.2 | 68.6±0.4 |
| LGM w/ DSA | 91.4±0.1 | 83.3±0.1 | 84.6±0.1 | 88.6±0.1 | 87.0±0.1 |
| CLP-DD (ours) | 89.2±0.1 | 83.9±0.1 | 79.2±0.1 | 87.6±0.0 | 85.0±0.1 |
| Full dataset | 95.1±0.1 | 89.3±0.2 | 92.4±0.0 | 94.0±0.1 | 92.7±0.1 |
| Method | DINOv2 | MoCo-v3 | CLIP | EVA-02 | Average |
|---|---|---|---|---|---|
| Random | 50.3±0.5 | 38.7±0.5 | 31.7±0.4 | 37.7±0.6 | 39.6±0.5 |
| Neighbor | 67.7±0.1 | 56.3±0.1 | 38.7±0.1 | 50.0±0.1 | 53.2±0.1 |
| Centroid | 69.5±0.0 | 57.4±0.1 | 53.8±0.1 | 58.0±0.0 | 59.6±0.1 |
| LGM w/ DSA | 75.0±0.1 | 63.2±0.0 | 62.9±0.0 | 70.3±0.1 | 67.8±0.1 |
| CLP-DD (ours) | 75.6±0.1 | 63.4±0.1 | 62.4±0.1 | 71.2±0.1 | 68.3±0.1 |
| Full dataset | 83.0±0.1 | 76.5±0.0 | 78.7±0.0 | 81.7±0.0 | 80.0±0.1 |
| Method | DINOv2 | MoCo-v3 | CLIP | EVA-02 | Average |
|---|---|---|---|---|---|
| Random | 57.1±8.6 | 67.4±5.7 | 71.9±4.5 | 59.8±13.3 | 64.0±8.0 |
| Neighbor | 67.3±5.8 | 45.2±11.4 | 74.6±6.5 | 74.3±3.3 | 65.3±6.8 |
| Centroid | 65.7±3.4 | 62.0±5.2 | 69.8±8.6 | 64.6±7.9 | 65.5±6.3 |
| LGM | 58.3±3.2 | 63.4±1.6 | 64.1±2.2 | 64.8±1.8 | 62.6±2.2 |
| LGM w/ DSA | 82.1±2.9 | 77.8±0.0 | 77.9±0.2 | 78.0±0.8 | 79.0±1.0 |
| CLP-DD (ours) | 80.0±2.3 | 78.5±0.4 | 75.4±0.3 | 79.6±0.4 | 78.3±0.8 |
| Full dataset | 95.5±0.1 | 74.3±0.1 | 86.0±0.1 | 90.4±0.2 | 86.5±0.1 |
| Outer objective | DINOv2 | MoCo-v3 | CLIP | EVA-02 | Average |
|---|---|---|---|---|---|
| MSE (w/o class anchor) | 88.3±0.0 | 82.1±0.1 | 74.1±0.1 | 84.1±0.1 | 82.1±0.1 |
| Class-anchor CE (ours) | 89.2±0.1 | 83.9±0.1 | 79.2±0.1 | 87.6±0.0 | 85.0±0.1 |
@article{peng2026clpdd,
title = {Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models},
author = {Peng, Bincheng and Li, Guang and Liu, Ping and Ogawa, Takahiro and Haseyama, Miki},
journal = {arXiv preprint arXiv:2605.07194},
year = {2026}
}