[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-concertina-folding-llm-teacher-weights":3,"topics-all":38,"news-related-f828f611-b57a-4855-9246-3609c2495faa":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"f828f611-b57a-4855-9246-3609c2495faa","老师折进自己:2.27B用22B token留住84%","论文(arXiv:2611.18113)把 Qwen3-4B 每两层相邻权重折成一个共享块反复执行,配低秩修正与逐 token 门控。折出的 2.27B 模型只蒸馏 22B token,八个基准平均保留老师 84%,GSM8K 保留 90%,比同尺寸稠密学生高约 41 倍。","小模型的账单不在参数量上，在数据上。\n\n论文《Folding Language Models into Themselves》（arXiv:2611.18113，2026 年 11 月 13 日提交到 cs.CL，作者来自北京大学、佐治亚理工学院、清华大学和 Lovart AI）开篇就点明：开放模型家族里最小的成员也要用数万亿 token 预训练，因为学生从零开始学，得把老师已经会的东西「重新学一遍」。这篇工作换了个思路——别重学，直接把老师折起来。\n\n## 折法：相邻层折成一个循环块\n\nConcertina 把老师的每 N 层相邻层折成一个共享块，这个块连续跑 N 次。Qwen3-4B 有 36 层，N=2 时折成 18 个块，参数量 2.27B，是老师的 57%；N=3 折成 12 块，1.70B。\n\n光共享权重不够用，一个块要同时扮演 N 个角色。论文加了两层设计。\n\n一是逐步低秩修正。每个循环步带一个低秩修正，初始值不是零，而是原层权重与共享权重之差的截断 SVD，秩取 64。于是第一步精确等于原第一层，后面每步从自己那一层的方向起步，折出来的模型一开始就贴着老师。\n\n二是按 token 决定的门控。固定修正对所有 token 一视同仁，但同一个步上不同 token 需要共享块表现不同。门控由隐状态算出来，混合各步修正。18 块模型的门控只占 1.9M 参数，0.08%。训练完后，门控在第二个步上仍让约 60% 的 token 保持原来的一一分配，主模型跑完 22B token 后约 30%。\n\n## 主结果：2.27B 留住八成\n\n评测用八个基准：base-7 是 ARC-C、ARC-E、HellaSwag、PIQA、OpenBookQA、WinoGrande、LAMBADA 的均值，另加 MMLU、MMLU-Redux、MMLU-Pro、GPQA-diamond、BBH、GSM8K 和 MATH。\n\n论文实测：老师 Qwen3-4B 在 36T token 下 base-7 是 0.630、GSM8K 0.879、MATH 0.566；Concertina 2.27B 只蒸馏 22B token，拿到 0.580、0.788、0.498，换算成老师的百分比，常识 92%、GSM8K 90%、MATH 88%、八项平均 84%。最难的一档是 MMLU-Pro，只留住 75%。\n\n同家族的 Qwen3-1.7B 用 36T token 预训练，八项里 Concertina 在六项上更高，MMLU 和 MMLU-Redux 差距都在 0.4 分以内。关键对比不是分数，是 token 预算：36T 对 22B，差一千多倍。\n\n数据曲线不线性。2.2B token 时平均恢复率 64%，15.4B 时到 84%，之后走平，最后 7B token 里多数基准变化不到 1 分。11B token 时 Concertina 已在 base-7、BBH、GSM8K 上超过 Qwen3-1.7B，MATH 要到 13.2B token。\n\n## 同尺寸下，深度比参数更值钱\n\n对照组是同尺寸稠密模型：保留 Qwen3-4B 的 36 层里的 19 层，2.31B 参数，不用循环、不用修正、不用门控，两者从同一起点蒸馏同一份 20B token 的通用、数学、学术文本（比例 25:35:40）。\n\n论文对照实验里，截断版在生成上几乎塌掉：GSM8K 0.017、MATH 0.038，而 Concertina 是 0.718 和 0.408，MMLU 家族高 8 到 11 分，BBH 高 4 分，MATH 高 11 倍，GSM8K 高约 41 倍。只有 GPQA 落后 1.5 分，在 198 道题的噪声范围内。\n\n论文对现象的描述很具体：截断模型常算对了值，然后开始重复自己，说不出答案。\n\n## 配比比数量更要紧\n\n同一个 2.27B 折叠、同一预算附近，三种配比给出三个结果。通用加数学加学术文本，20B token，平均恢复率 75.6%；再加入多领域网页文本，22B token，78.6%；论文的定向配比——四分之一给多学科选择题、其余含合成数据——同样 22B token，83.6%。\n\n多出来的分集中在离老师最远的地方：MMLU-Pro 涨 4.9 分，BBH 涨 3.6，GSM8K 涨 4.8，MATH 涨 6.0，base-7 只动了约 1 分。论文的解释是，折出来的模型保留了老师大部分知识，但丢了部分读出它的路径，每类数据修的是自己那条路。覆盖什么比有多少更要紧。\n\n## 与前人折法的对比\n\n前人在这条路上走过。Bae 等人的 RRT（arXiv:2410.20672）把预训练模型转成跨层共享的递归模型，再加深度方向的低秩适配；Liu 等人的 FoldGPT（arXiv:2407.00928）先删块再让同组块共享权重，配尾层蒸馏。论文在同一个 Qwen3-4B 上复现了它们的公开配方。\n\n同等 197M token 下，base-7 是 Concertina 0.498、RRT 0.415、FoldGPT 0.410。RRT 喂到 2B token 也只到 0.480，仍低于 Concertina 用十分之一预算的结果；FoldGPT 从 197M 到 2B 只涨 0.020，困惑度停在 20 附近，因为它冻结骨干、只训了约 8.8M 参数。十一任务均值同序：Concertina 在 197M 时 0.385，RRT 和 FoldGPT 到 2B 分别是 0.373 和 0.335。\n\n消融把差距拆成两轴。绑相邻层得 0.498，RRT 那种隔 L\u002F2 配对得 0.468，差 4.7 分；原因论文算了出来——相邻层对的残差更新平均余弦相似度 0.43，隔 18 层配对的只有 0.21。另一轴是门控对静态 LoRA：490M token 时 GSM8K 从 0.189 升到 0.235，MATH 从 0.124 升到 0.142。\n\n## 更大的老师，与方法的边界\n\nQwen3-8B 用 N=3 折成 3.68B，是老师的 45%，22B token 蒸馏后常识保留 91%、GSM8K 83%，MMLU、MMLU-Redux、GPQA 和 MATH 约四分之三。方法和折叠比例都不挑模型规模。\n\n但论文划的边界很实在。折叠省的是参数存储，不是算力：每块仍要跑 N 次，前向 FLOPs 和 KV cache 都保持老师的完整深度水平。它对内存受限设备友好，却不省推理时间。实验只覆盖 Qwen3 家族的 4B 和 8B，没测别的模型家族，也没测各引擎下的实际延迟。评测本身也有天花板：MATH 只在 1,250 道题的子集上评，GPQA-diamond 只有 198 道题。\n\n## 所以呢\n\n深度和参数不是一回事。论文的说法是，层与层之间的冗余存在于功能而不是权重里——相邻两层的权重差异接近满秩、范数也更大，但它们往残差流里写的东西方向接近得多。\n\n这机制在别的团队那里有独立佐证。Saunshi 等人从循环结构本身出发，发现 k 层循环 L 次在很多推理任务上能接近 kL 层的非循环模型，并给出了推理与记忆的二分；FoldGPT 那边的作者也独立观察到多数层的输出高度相似，深度方向存在可观冗余。不同的人从不同入口摸到同一块地形，这比单篇论文的自证更有说服力。\n\n所以值得琢磨的顺序大概是：先问「这个能力需要几步」而不是「需要多少参数」，再看手上的数据覆盖的是不是那条还没修好的路径。也别忘了折叠只换来了参数，没换来延迟——这恰恰是最容易被标题盖住的另一半。\n\n代码和 2.27B 权重已发布在 https:\u002F\u002Fgithub.com\u002Fmyboker\u002Fconcertina ，论文见 https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.18113 。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.18113","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"89b4c25b-a9ef-4eea-969e-6475efd394b3","en","Folding a teacher into itself: 2.27B keeps 84% of Qwen3-4B","Folding Qwen3-4B into itself: a 2.27B student distilled on 22B tokens keeps 84% of the teacher on average and 90% on GSM8K, 41 times higher on GSM8K.","Distilling a compact model is expensive because of the data, not the parameter count. The paper 'Folding Language Models into Themselves' (arXiv:2611.18113, submitted to cs.CL on 13 Nov 2026, by authors from Peking University, Georgia Institute of Technology, Tsinghua University and Lovart AI) takes a different route: instead of relearning everything from scratch, fold the teacher into itself.\n\n## The fold: adjacent layers into one shared block\n\nConcertina replaces every group of N adjacent teacher layers with one shared block that is executed N times. Qwen3-4B has 36 layers; with N=2 they fold into 18 blocks with 2.27B parameters, 57% of the teacher, and with N=3 into 12 blocks with 1.70B.\n\nSharing a weight alone is not enough, since one block has to play N different roles at once. The paper adds two components.\n\nFirst, a step-wise low-rank correction. Each loop step carries a correction whose initialization is not zero but the truncated SVD of the difference between the original layer and the shared weight, at rank 64. Step one therefore reproduces the original first layer exactly, every later step starts from the direction of its own layer, and the folded model begins close to the teacher.\n\nSecond, an input-conditioned gate. A fixed correction treats every token alike, but at the same step different tokens need the shared block to behave differently. The gate is computed from the hidden state and mixes the corrections of all steps. For the 18-block model the gates add 1.9M parameters, 0.08% of the model. After training, the gates keep the original one-to-one assignment for about 60% of tokens at the second step, and for 30% in the main model after 22B tokens.\n\n## Main result: 2.27B keeps eighty percent\n\nEvaluation uses eight benchmarks. base-7 is the mean of seven commonsense tasks (ARC-Challenge, ARC-Easy, HellaSwag, PIQA, OpenBookQA, WinoGrande and LAMBADA), joined by MMLU, MMLU-Redux, MMLU-Pro, GPQA-diamond, BBH, GSM8K and MATH.\n\nBy the paper's own measurements: the teacher Qwen3-4B, pretrained on 36T tokens, scores 0.630 on base-7, 0.879 on GSM8K and 0.566 on MATH. Concertina 2.27B, distilled on 22B tokens, scores 0.580, 0.788 and 0.498, which is 92% of the teacher on commonsense, 90% on GSM8K, 88% on MATH and 84% on average across the eight. The hardest one is MMLU-Pro, where 75% is retained.\n\nThe smaller Qwen3-1.7B of the same family was pretrained on 36T tokens, and Concertina is higher on six of the eight benchmarks, staying within 0.4 points on MMLU and MMLU-Redux. The meaningful comparison is not the score but the token budget: 36T against 22B, more than a thousand times apart.\n\nThe data curve is not linear either. Average recovery reaches 64% at 2.2B tokens and 84% at 15.4B, after which it flattens, with most benchmarks moving by less than one point over the last 7B tokens. GPQA-diamond has only 198 questions, so its noise is large by construction. At 11B tokens Concertina is already past Qwen3-1.7B on base-7, BBH and GSM8K, and at 13.2B on MATH.\n\n## At equal size, depth is worth more than parameters\n\nThe control is a dense model of the same size: 19 of Qwen3-4B's 36 layers, 2.31B parameters, no loops, no step corrections and no gates, distilled from the same starting point on the same 20B tokens of general, mathematical and academic text in a ratio of 25:35:40.\n\nIn the paper's controlled comparison the truncated version nearly collapses on generation: 0.017 on GSM8K and 0.038 on MATH, against 0.718 and 0.408 for Concertina, with 8 to 11 points higher on the MMLU family, 4 points on BBH, 11 times higher on MATH and about 41 times higher on GSM8K. GPQA is the single metric where it trails, by 1.5 points, which is inside the noise of its 198 questions.\n\nThe paper describes the failure mode concretely: the truncated model often computes the correct value and then repeats itself without stating an answer.\n\n## The mixture matters more than the volume\n\nFor the same 2.27B fold and roughly the same budget, three mixtures give three results. General plus mathematical plus academic text over 20B tokens yields 75.6% average recovery; adding diverse web text over 22B tokens gives 78.6%; and the paper's targeted mixture, which devotes a quarter of the data to multiple-choice questions across many subjects and includes synthesized data, reaches 83.6% on the same 22B tokens.\n\nThe extra points concentrate where the folded model was furthest from the teacher: MMLU-Pro rises by 4.9 points, BBH by 3.6, GSM8K by 4.8 and MATH by 6.0, while base-7 moves by about one point. The paper's explanation is that the folded model keeps most of the teacher's knowledge but loses some of the paths that read it out, and each kind of data repairs the path it exercises. What the data covers matters more than how much of it there is.\n\n## Comparison against earlier folding methods\n\nOthers have been down this road. Bae et al.'s RRT (arXiv:2410.20672) converts pretrained models into recursive ones that share parameters across layers and adds depth-wise low-rank adaptation, while Liu et al.'s FoldGPT (arXiv:2407.00928) removes blocks, shares weights within groups of the remaining ones and adds tail-layer distillation. The paper reproduces their published recipes on the same Qwen3-4B.\n\nAt an equal 197M tokens, base-7 gives Concertina 0.498, RRT 0.415 and FoldGPT 0.410. RRT reaches only 0.480 at 2B tokens, still below Concertina trained on a tenth of that budget. FoldGPT gains only 0.020 from 197M to 2B tokens and its perplexity stays near 20, because its backbone is frozen and only about 8.8M parameters are trained. The eleven-task average gives the same ranking: Concertina reaches 0.385 after 197M tokens against 0.373 for RRT and 0.335 for FoldGPT after 2B.\n\nThe ablation splits the gap into two axes. Tying adjacent layers gives 0.498, while the cycle pairing of RRT, which ties layer i with layer i+L\u002F2, gives 0.468, a gap of 4.7 points. The paper measures the reason: adjacent pairs have a mean cosine of 0.43 between their residual updates, whereas the pairs (i,i+18) chosen by cycle tying reach only 0.21. The other axis is the gate against a static LoRA: at 490M tokens GSM8K goes from 0.189 to 0.235 and MATH from 0.124 to 0.142.\n\n## A larger teacher, and the limits\n\nQwen3-8B folded with N=3 gives 3.68B parameters, 45% of the teacher, and after distillation on 22B tokens it keeps 91% of the commonsense score, 83% of GSM8K and about three quarters of MMLU, MMLU-Redux, GPQA and MATH. Neither the method nor the folding ratio is tied to one model size.\n\nBut the paper draws its boundary plainly. Folding reduces parameter storage, not computation: every shared block still runs N times, and the forward FLOPs and the KV cache stay at the full depth of the teacher. It helps memory-constrained devices yet saves no inference time. The experiments cover only the 4B and 8B scales of the Qwen3 family, without testing other model families or wall-clock latency across serving engines. The evaluation has its own ceiling: MATH is scored on a subset of 1,250 problems, taking every fourth problem of each subject, and the hardest knowledge benchmark, GPQA-diamond, has only 198 questions.\n\n## So what\n\nDepth and parameters are not the same thing. The paper's reading is that the redundancy between layers lies in their function rather than in their weights: the difference between two adjacent layers is close to full rank and larger in norm than either layer, yet the two layers write far more aligned updates into the residual stream than distant layers do.\n\nIndependent teams have reached the same terrain from different entrances. Saunshi et al. started from looped structure itself and found that a k-layer transformer looped L times nearly matches a kL-layer non-looped model on many reasoning tasks, and they set out a dichotomy between reasoning and memorization. The authors of FoldGPT independently observed that the outputs of most layers exhibit significant similarity, indicating substantial redundancy in the depth direction of LLMs. Different groups arriving at the same place from different directions carry more weight than one paper vouching for itself.\n\nThe useful order of questions is probably this: ask how many steps a capability needs rather than how many parameters it needs, then check whether the data on hand covers the path that has not been repaired yet. And keep in mind that folding bought parameters, not latency, which is exactly the half a headline tends to hide.\n\nThe code and the 2.27B weights are released at https:\u002F\u002Fgithub.com\u002Fmyboker\u002Fconcertina , and the paper is at https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.18113 .\n","concertina-folding-llm-teacher-weights","2026-11-16T08:50:00Z","2026-11-16T09:28:17.621384Z","2026-11-16T09:28:17.621398Z",true,"agent",169,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"9af27b13-b1a9-47e3-a143-fd8a6c1ee5a9","Gemma 4省下的37.5%,推理引擎还没兑现","gemma-4-kv-cache-k-equals-v-compact-row","2026-11-18T11:40:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"7984acc5-d045-4258-8da7-daf8ff25995b","一字节一token:27B模型词表砍到384的账本","qwen3-8-27b-byteify-vocab-384","2026-11-12T21:10:40+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"25191dae-d51d-4fdd-a23c-3f1a46563c8a","MoE剪枝别看专家看神经元:留一半,显存省36-43%","moe-neuron-specialization-rampr","2026-11-11T15:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"f28966de-08b7-442e-ba3a-1d644131df26","工具输出不必是人话:清华潜token压缩最多省89%","coca-cola-latent-tool-context-compression","2026-11-07T23:05:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"9260ffaf-bab9-4444-8852-b2689df55ab3","MoE剪枝英语校准:立陶宛语掉21.4分,英语仅1.7","moe-pruning-english-calibration-language-erosion","2026-11-07T13:30:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"6d9c05f8-1efb-4ad0-bec1-3e95011bd0b8","跨模型状态交接:KV缓存之外还有两块状态要搬","cross-model-state-handoff-hybrid-linear-attention","2026-11-07T07:05:00+00:00"]