[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-cherry-selective-token-training":3,"news-related-4f19e049-42e2-43f3-9a9e-4ff97cda00dd":31},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":24,"published_at":25,"created_at":26,"modified_at":27,"is_published":28,"publish_type":29,"image_url":13,"view_count":30},"4f19e049-42e2-43f3-9a9e-4ff97cda00dd","CHERRY 用「15% 监督 + 6 层折叠 + 专家融合」三件套把 LLM 训练推到新性价比边界","2026 年 6 月 30 日，Kwon 与 Park 在 arXiv 上放出 CHERRY（Compressed Hierarchical Experts with Recurrent Representational Yield）预印本，把 LLM 训练效率拆成三条独立可叠加的路径，并在韩语基础模型 CHERRY-1.8B 上端到端跑通。\n\n第一条是 Selective Ground Truth Token Training（SGT）：监督只打在输出 token 中约 15% 的「语义有效 token」上，剩下的 85% 完全不计算 loss，但作者证明只要位置共享的 Transformer 权重上存在正梯度耦合（γ̅ = 0.72），那些无监督 token 的损失仍会自然下降——这等价于单 token 监督效率提升 4.5×，而效果在打乱顺序的文本上彻底消失，说明这是自然语言结构本身的红利，而非优化器的副作用。\n\n第二条是「深度压缩 + 循环恢复」：把 1B 参数、48 层的密集 Transformer 通过相邻层平均折叠到 6 层（227M），再让模型以学得的循环展开把有效深度补回 34 层，最终 held-out loss 2.934 落在 566M 密集模型 2.926 的测量噪声内，参数直接砍掉 2.5×。第三条则是把若干压缩模型拼成 Mixture of Efficient Experts（MoEE）+ 多 token 预测，2-expert MoEE 拿到 2.789 loss，比单压缩模型的 2.926 还能再低一截。\n\n三条路径彼此正交、彼此可堆叠：SGT 改监督信号、深度压缩改架构、MoEE 改推理时路由，对应在 CHERRY-1.8B 上，1.8B 总参数下达到了与「标准全监督 2B+ 模型」可比的水平。整套方法给的是一份「在数据\u002F算力紧约束下怎么把 LLM 训练做出可解释性价比」的工程范本：与其继续靠堆参数换能力，不如先证明监督信号、深度、有效容量三件事可以各自打折再乘起来——这条路径在韩语、阿拉伯语等中等资源语言上，会比在英语开源模型上更早看到落地回报。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.31796","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",[],"cherry-selective-token-training","2026-07-01T22:00:00Z","2026-07-01T22:16:41.382969Z","2026-08-19T02:08:40.142862Z",true,"agent",132,{"items":32},[33,38,43,48,53,58],{"id":34,"title":35,"news_slug":36,"published_at":37},"5878a668-282c-4b88-b2b8-7eef40b7938c","LFM2.5-2.6B：2.5GB 内存跑本机 Agent 220 tok\u002Fs","lfm2-5-2-6b-on-device-agent","2026-08-11T00:00:00+00:00",{"id":39,"title":40,"news_slug":41,"published_at":42},"d2c56430-f9a7-4844-bfab-a8651279a70c","ResKV 不再把 KV 缓存压缩等同于删词：给被淘汰的信息留一份残差账本","reskv-residual-kv-cache-compression","2026-08-03T10:43:23+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"d04db7ae-1027-4ba5-8939-563fd7372c21","Iterative Puzzle：Nemotron-3 砍到 62%，吞吐 2.03×","nvidia-iterative-puzzle-nemotron-3-super","2026-07-18T06:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"06651fbd-69a7-42b7-adac-68fc5db5063e","Soofi S 30B 用 MoE + 混合架构挤进完全开源头名:德国把主权 AI 写进 3.2B 激活参数","soofi-s-30b-sovereign","2026-07-13T20:04:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"5a5b1531-e1b2-469b-8064-772223231183","KronQ：Kronecker Hessian 拆掉 GPTQ 的 2-bit 墙","kronq-kronecker-hessian-gptq","2026-07-13T16:02:00+00:00"]