Importance-Aware Low-Rank Distillation of Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SVD) is a principled tool for parameter reduction, evidence from large language models (LLMs) suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks.

Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective.

We apply SVDtrunc to FLUX.dev across compression levels ranging from 40% to 90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models.

Key finding

Low-rank approximation fails on language models. On Diffusion Transformers it works.

In autoregressive LLMs, naively truncating the singular values of weight matrices is known to degrade quality severely, and can collapse the model outright unless it is propped up by careful fine-tuning or architectural adaptation. Diffusion Transformers behave in the opposite way. Quality falls off smoothly as rank is removed, the exploitable redundancy is spread thinly across projection matrices throughout the whole network rather than concentrated in a few blocks, and the tolerance is already present in the pretrained weights before any fine-tuning takes place. Low-rank compression is therefore a practical tool for large text-to-image models in a way it never became for language models.

Mean quality loss at 68% of the parameters

0.75%

Best competing method: 5.55%

Averaged over GenEval, HPSv2 and DPG, measured against full FLUX.dev.

Mean quality loss at 70% of the parameters, with no fine-tuning at all

9.86%

Block pruning: 94.52%. EcoDiff: 85.50%.

Both baselines collapse entirely at this budget while SVDtrunc still generates usable images, so the tolerance is intrinsic to the pretrained model rather than an artefact of our distillation.

Damage from compressing one block, against removing it

5× smaller

Mean CMMD 0.23 against 1.18

Probing each block in turn shows redundancy sits inside the projection matrices, not in whole blocks. Removing a block destroys function the rest must compensate for.

Mean quality loss on 4-step FLUX.schnell at 60% of the parameters

4.53%

A model already distilled along the time axis still compresses gracefully, so the redundancy is structural rather than a by-product of multi-step sampling.

Compression costs memorised instances before it costs image quality

What the model gives up first is not fidelity but specificity. Under stronger budgets it stops reproducing particular memorised subjects, so the Leaning Tower of Pisa, Bugs Bunny and the Minions become well-rendered images of the general concept that have lost the distinctive identity. Stylised domains drift in parallel, with anime, manga and cartoon renderings pulling towards photorealism. Benchmark scores stay relatively stable through both effects.

That points at what the removed parameters were doing: part of the capacity in a 12 B DiT appears to store highly specific concepts rather than the reusable visual features and compositional patterns that general image synthesis needs. Step through both effects below.

Bugs Bunny generated by uncompressed FLUX.dev

At 39% of the parameters, a generic yellow and red cartoon rabbit character instead of Bugs Bunny

Manga ink portrait generated by uncompressed FLUX.dev

At 39% of the parameters, the same manga portrait rendered in a more realistic illustrative style

Method

Probe, then allocate rank by importance

Compress projections, do not remove blocks

Existing parameter reduction for DiTs prunes or substitutes entire Transformer blocks. We build on two hypotheses instead. Individual blocks are not fully redundant, so removing one destroys functionality the rest must compensate for. And blocks do not contribute equally, so the degree of reduction should vary across the network.

For a linear projection \(W \in \mathbb{R}^{n\times m}\) we keep only the top \(r\) singular values, factorising it into two smaller projections:

\[ W = U\Sigma V^{T} \;\approx\; U_r \Sigma_r V_r^{T} = BA, \qquad B = U_r\sqrt{\Sigma_r},\quad A = \sqrt{\Sigma_r}V_r^{T} \]

The square roots are split across both factors for numerical stability. A target compression ratio \(c\) then fixes the rank, since the original matrix holds \(P_0 = mn\) parameters and the factorised form holds \(P_r = r(m+n)\):

\[ r \le \frac{(1-c)\,nm}{n+m} \]

Every projection inside a block shares that block's ratio, while the surrounding normalisations and non-linearities are left untouched. The architecture is unchanged and the block's internal interactions survive.

Ranking blocks by compression probing

We caption 1000 LAION images with JoyCaption and generate one image per caption with the uncompressed model. We then build one probed model per block, compressing only that block at \(c_m = 0.6\), and regenerate with identical prompts and seeds.

Because compressing a block can change pose or appearance without lowering quality, pairwise metrics such as LPIPS are misleading here. We rank with CMMD, a distribution-level metric, preferring it over FID for stability at smaller sample counts.

Blocks are ordered by increasing importance and receive a linearly decreasing ratio, with the slope \(\alpha\) chosen to hit the global parameter budget and blocks past a cut-off left entirely uncompressed:

\[ c_i = c_m - \alpha\,\mathrm{order}(i) \]

Fine-tuning with modular knowledge distillation

Because the block structure is preserved, the student can be aligned to the teacher module by module, not just at the output. We combine the native rectified-flow objective with velocity-level distillation and per-block feature matching:

\[ \mathcal{L} = \lambda_\text{RF}\mathcal{L}_\text{RF} + \lambda_\text{F}\mathcal{L}_\text{F} + \lambda_\text{KD}\mathcal{L}_\text{KD} \]

with \(\lambda_i\) equalising the magnitude of each block's contribution.

Generated images after probing double-stream blocks with pruning (top row) versus truncated SVD (bottom row). The pruned outputs are severely degraded or destroyed; the SVD outputs stay close to the original. — **Pruning against compression.** Probing double-stream blocks by pruning *(top)* and by truncated SVD *(bottom)*. Pruning causes severe degradation and, for early blocks, catastrophic failure. Mean CMMD per probed block is 1.18 ± 3.36 for pruning against 0.23 ± 0.65 for compression.

Two image pairs showing that compressing a block can change pose and appearance without reducing image quality. — **Why a distribution metric.** Compressing a block can shift pose or appearance *(right)* relative to the original *(left)* without lowering quality, which makes pixel-wise similarity a poor ranking signal.

Results

Comparison to the state of the art

**Compression of FLUX.dev.** Rank is based on average quality reduction R. Our medium model takes first place, and our small model is runner-up with substantially fewer parameters than every competing approach.
Method	Venue	Params % ↓	VRAM % ↓	GenEval ↑	HPSv2 ↑	DPG ↑	R % ↓	Rank ↓
FLUX.dev	teacher	100	100.0	0.647	31.70	83.9	0.00	n/a
Chroma-HD	HF	75	82.5	0.593	n/a	84.0	4.09	3
FluxLite	HF '24	68	78.8	0.523	31.31	79.3	8.61	5
TinyFusion	CVPR '25	68	74.4	0.511	n/a	77.2	14.48	6
Dense2MoE	ICCV '25	75	n/a	0.403	n/a	73.6	24.97	9
FastFlux	AAAI '26	80	n/a	0.530	27.26	n/a	16.04	8
PPCL	CVPR '26	68	74.4	0.605	n/a	80.0	5.55	4
Hier.Prune	AAAI '26	68	74.4	0.503	n/a	75.7	15.99	7
EcoDiff	ICLR '26	80	n/a	0.399	25.99	n/a	28.17	10
SVDtrunc-m	this work	68	78.2	0.645	31.27	83.4	0.75	1
SVDtrunc-s	this work	57	70.7	0.616	31.22	82.6	2.60	2

Numbers for TinyFusion, Dense2MoE, Chroma-HD and HierarchicalPrune are taken from PPCL, as no public checkpoints or inference code are available. All other models are re-evaluated under one protocol: 1024×1024, 50 steps, CFG 3.5, fixed seeds per prompt.

Training-free compression

Applying the importance-guided schedule once, with no fine-tuning, isolates how much of the result is intrinsic to the pretrained model. At a 90% budget, where barely any compression has taken place, importance-ranked block pruning is still marginally ahead of us. From 80% downward the ordering inverts sharply, and by 70% both pruning and EcoDiff have collapsed entirely while SVDtrunc is still generating usable images.

**Training-free compression.** No fine-tuning of any method. SVDtrunc holds together where pruning and EcoDiff collapse.
Method	P % ↓	GenEval ↑	HPSv2 ↑	DPG ↑	R % ↓
FLUX.dev	100	0.647	31.70	83.9	0.00
EcoDiff	90	0.452	29.67	78.0	14.52
Pruning	90	0.646	31.02	83.3	1.00
SVDtrunc	90	0.638	29.03	78.9	5.24
EcoDiff	80	0.187	23.63	49.1	45.99
Pruning	80	0.381	24.88	65.2	28.31
SVDtrunc	80	0.617	28.86	78.4	6.69
EcoDiff	70	0.002	7.26	17.0	85.50
Pruning	70	0.000	4.37	2.2	94.52
SVDtrunc	70	0.580	28.48	76.6	9.86

Applied to a step-distilled model

One might expect a model already distilled along the temporal axis to have less redundancy left to give. FLUX.schnell generates in four steps, and compresses about as gracefully as FLUX.dev does.

**Temporally-distilled FLUX.schnell.** Degradation stays gradual, with no abrupt failure.
Method	P % ↓	GenEval ↑	HPSv2 ↑	DPG ↑	R % ↓
FLUX.schnell	100	0.667	30.85	85.0	0.00
SVDtrunc	90	0.666	29.59	85.2	1.34
SVDtrunc	80	0.668	29.48	84.9	1.48
SVDtrunc	70	0.648	29.30	84.8	2.72
SVDtrunc	60	0.621	29.05	84.3	4.53

Does importance-aware allocation matter?

All three schedules degrade smoothly, which is itself evidence that redundancy is broadly distributed, since even a random allocation works reasonably. The importance ranking earns its keep as the budget tightens.

**Compression schedule ablation.** Uniform, random and importance-aware allocation, each after fine-tuning.
Schedule	P % ↓	GenEval ↑	HPSv2 ↑	DPG ↑	R % ↓
FLUX.dev	100	0.647	31.70	83.9	0.00
Random	68	0.633	31.19	82.9	1.63
Uniform	68	0.633	31.38	82.8	1.46
Importance-aware	68	0.645	31.27	83.4	0.75
Random	57	0.588	30.22	81.1	5.68
Uniform	57	0.613	30.92	82.3	3.19
Importance-aware	57	0.616	31.22	82.6	2.60
Random	49	0.542	29.16	79.8	9.69
Uniform	49	0.582	29.96	81.2	6.23
Importance-aware	49	0.587	31.01	81.7	4.68

Practical details

Backbone: FLUX.dev, 12 B DiT parameters
Blocks compressed: |𝕊| = 30
Probing: 1000 LAION images, JoyCaption prompts, CMMD ranking
Fine-tuning: 40k steps, batch 4, Adafactor, lr 1.8×10⁻⁶, one H200
Training data: 190k FLUX.dev generations
Cost: about 175 H200 h probing (parallelisable), plus 126 h for ‑m and 63 h for ‑s
Step runtime: 424 ms to 380 ms (‑m) and 354 ms (‑s)
With FP8: R rises from 0.75 to 1.87 (‑m) and 2.60 to 3.57 (‑s)
Also tested: PixArt-Σ (0.6 B), same trend

Diagram of a multi-modal diffusion transformer block. Projection modules are highlighted green as the compressed components; normalisations and parameter-free operations are grey and left unchanged. — **Where the compression is applied.** Inside an MMDiT block, green modules are the projections we compress with block-wise low-rank approximation. Grey components hold negligible or no parameters and are left unchanged, preserving the architecture.