Distilling a compact model is expensive because of the data, not the parameter count. The paper 'Folding Language Models into Themselves' (arXiv:2611.18113, submitted to cs.CL on 13 Nov 2026, by authors from Peking University, Georgia Institute of Technology, Tsinghua University and Lovart AI) takes a different route: instead of relearning everything from scratch, fold the teacher into itself.
The fold: adjacent layers into one shared block
Concertina replaces every group of N adjacent teacher layers with one shared block that is executed N times. Qwen3-4B has 36 layers; with N=2 they fold into 18 blocks with 2.27B parameters, 57% of the teacher, and with N=3 into 12 blocks with 1.70B.
Sharing a weight alone is not enough, since one block has to play N different roles at once. The paper adds two components.
First, a step-wise low-rank correction. Each loop step carries a correction whose initialization is not zero but the truncated SVD of the difference between the original layer and the shared weight, at rank 64. Step one therefore reproduces the original first layer exactly, every later step starts from the direction of its own layer, and the folded model begins close to the teacher.
Second, an input-conditioned gate. A fixed correction treats every token alike, but at the same step different tokens need the shared block to behave differently. The gate is computed from the hidden state and mixes the corrections of all steps. For the 18-block model the gates add 1.9M parameters, 0.08% of the model. After training, the gates keep the original one-to-one assignment for about 60% of tokens at the second step, and for 30% in the main model after 22B tokens.
Main result: 2.27B keeps eighty percent
Evaluation uses eight benchmarks. base-7 is the mean of seven commonsense tasks (ARC-Challenge, ARC-Easy, HellaSwag, PIQA, OpenBookQA, WinoGrande and LAMBADA), joined by MMLU, MMLU-Redux, MMLU-Pro, GPQA-diamond, BBH, GSM8K and MATH.
By the paper's own measurements: the teacher Qwen3-4B, pretrained on 36T tokens, scores 0.630 on base-7, 0.879 on GSM8K and 0.566 on MATH. Concertina 2.27B, distilled on 22B tokens, scores 0.580, 0.788 and 0.498, which is 92% of the teacher on commonsense, 90% on GSM8K, 88% on MATH and 84% on average across the eight. The hardest one is MMLU-Pro, where 75% is retained.
The smaller Qwen3-1.7B of the same family was pretrained on 36T tokens, and Concertina is higher on six of the eight benchmarks, staying within 0.4 points on MMLU and MMLU-Redux. The meaningful comparison is not the score but the token budget: 36T against 22B, more than a thousand times apart.
The data curve is not linear either. Average recovery reaches 64% at 2.2B tokens and 84% at 15.4B, after which it flattens, with most benchmarks moving by less than one point over the last 7B tokens. GPQA-diamond has only 198 questions, so its noise is large by construction. At 11B tokens Concertina is already past Qwen3-1.7B on base-7, BBH and GSM8K, and at 13.2B on MATH.
At equal size, depth is worth more than parameters
The control is a dense model of the same size: 19 of Qwen3-4B's 36 layers, 2.31B parameters, no loops, no step corrections and no gates, distilled from the same starting point on the same 20B tokens of general, mathematical and academic text in a ratio of 25:35:40.
In the paper's controlled comparison the truncated version nearly collapses on generation: 0.017 on GSM8K and 0.038 on MATH, against 0.718 and 0.408 for Concertina, with 8 to 11 points higher on the MMLU family, 4 points on BBH, 11 times higher on MATH and about 41 times higher on GSM8K. GPQA is the single metric where it trails, by 1.5 points, which is inside the noise of its 198 questions.
The paper describes the failure mode concretely: the truncated model often computes the correct value and then repeats itself without stating an answer.
The mixture matters more than the volume
For the same 2.27B fold and roughly the same budget, three mixtures give three results. General plus mathematical plus academic text over 20B tokens yields 75.6% average recovery; adding diverse web text over 22B tokens gives 78.6%; and the paper's targeted mixture, which devotes a quarter of the data to multiple-choice questions across many subjects and includes synthesized data, reaches 83.6% on the same 22B tokens.
The extra points concentrate where the folded model was furthest from the teacher: MMLU-Pro rises by 4.9 points, BBH by 3.6, GSM8K by 4.8 and MATH by 6.0, while base-7 moves by about one point. The paper's explanation is that the folded model keeps most of the teacher's knowledge but loses some of the paths that read it out, and each kind of data repairs the path it exercises. What the data covers matters more than how much of it there is.
Comparison against earlier folding methods
Others have been down this road. Bae et al.'s RRT (arXiv:2410.20672) converts pretrained models into recursive ones that share parameters across layers and adds depth-wise low-rank adaptation, while Liu et al.'s FoldGPT (arXiv:2407.00928) removes blocks, shares weights within groups of the remaining ones and adds tail-layer distillation. The paper reproduces their published recipes on the same Qwen3-4B.
At an equal 197M tokens, base-7 gives Concertina 0.498, RRT 0.415 and FoldGPT 0.410. RRT reaches only 0.480 at 2B tokens, still below Concertina trained on a tenth of that budget. FoldGPT gains only 0.020 from 197M to 2B tokens and its perplexity stays near 20, because its backbone is frozen and only about 8.8M parameters are trained. The eleven-task average gives the same ranking: Concertina reaches 0.385 after 197M tokens against 0.373 for RRT and 0.335 for FoldGPT after 2B.
The ablation splits the gap into two axes. Tying adjacent layers gives 0.498, while the cycle pairing of RRT, which ties layer i with layer i+L/2, gives 0.468, a gap of 4.7 points. The paper measures the reason: adjacent pairs have a mean cosine of 0.43 between their residual updates, whereas the pairs (i,i+18) chosen by cycle tying reach only 0.21. The other axis is the gate against a static LoRA: at 490M tokens GSM8K goes from 0.189 to 0.235 and MATH from 0.124 to 0.142.
A larger teacher, and the limits
Qwen3-8B folded with N=3 gives 3.68B parameters, 45% of the teacher, and after distillation on 22B tokens it keeps 91% of the commonsense score, 83% of GSM8K and about three quarters of MMLU, MMLU-Redux, GPQA and MATH. Neither the method nor the folding ratio is tied to one model size.
But the paper draws its boundary plainly. Folding reduces parameter storage, not computation: every shared block still runs N times, and the forward FLOPs and the KV cache stay at the full depth of the teacher. It helps memory-constrained devices yet saves no inference time. The experiments cover only the 4B and 8B scales of the Qwen3 family, without testing other model families or wall-clock latency across serving engines. The evaluation has its own ceiling: MATH is scored on a subset of 1,250 problems, taking every fourth problem of each subject, and the hardest knowledge benchmark, GPQA-diamond, has only 198 questions.
So what
Depth and parameters are not the same thing. The paper's reading is that the redundancy between layers lies in their function rather than in their weights: the difference between two adjacent layers is close to full rank and larger in norm than either layer, yet the two layers write far more aligned updates into the residual stream than distant layers do.
Independent teams have reached the same terrain from different entrances. Saunshi et al. started from looped structure itself and found that a k-layer transformer looped L times nearly matches a kL-layer non-looped model on many reasoning tasks, and they set out a dichotomy between reasoning and memorization. The authors of FoldGPT independently observed that the outputs of most layers exhibit significant similarity, indicating substantial redundancy in the depth direction of LLMs. Different groups arriving at the same place from different directions carry more weight than one paper vouching for itself.
The useful order of questions is probably this: ask how many steps a capability needs rather than how many parameters it needs, then check whether the data on hand covers the path that has not been repaired yet. And keep in mind that folding bought parameters, not latency, which is exactly the half a headline tends to hide.
The code and the 2.27B weights are released at https://github.com/myboker/concertina , and the paper is at https://arxiv.org/abs/2611.18113 .