[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-echo-baidu-multi-token-prediction-self-recursion":3,"topics-all":38,"news-related-addfda39-d64f-4cb9-809a-7dfadd72d421":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"addfda39-d64f-4cb9-809a-7dfadd72d421","ECHO:百度把多Token预测搬进主干最后一层","百度ERNIE团队的论文ECHO指出,现有多Token预测(MTP)只是挂在主干上的辅助模块,监督信号被这些模块吸收,主干得不到直接训练。ECHO让主干最后一层递归预测所有未来位置,配合动态深度采样,把摊薄的MTP训练开销压到1.5个深度1步,并让模型成为自己的草稿器,在开放式高熵生成上仍保持草稿与目标的对齐。","投机解码几乎成了推理优化的默认配置：用小模型先草拟几个 token，再让大模型一次性验证。但这条路有个隐含前提——草稿器要么训练时顺带学出来，要么事后单独微调。如果两条都不可行，还有第三条路吗？百度 ERNIE 团队 10 月 17 日挂到 arXiv 的论文 ECHO 试图证明：主干模型自己就能当自己的草稿器（arXiv:2610.24766）。\n\n## 辅助模块吃掉了监督信号\n\n论文把矛头对准多 Token 预测（MTP）。并行 MTP 从同一个 hidden state 独立预测多个未来位置，忽略因果依赖；因果 MTP（如 DeepSeek-V3）通过移位 token embedding 保留自回归依赖，预测质量明显更好。但两者共享一个结构性问题——未来 token 由挂在主干上的专用模块完成，而非融入自回归生成本身。\n\n代价有两个。其一是间接监督：MTP 的 loss 主要压在辅助模块上，主干只能间接接收到多 horizon 信号。其二是长 horizon 太贵：每加一层就多一份显存里的模块、多一次计算。这条约束直接把大规模因果 MTP 预训练钉在单个草稿 token 上——DeepSeek-V3 只用一个 MTP 模块，小米的 MiMo 则发现继续加 MTP 层不再有提升。\n\nECHO 与\"循环 Transformer\"听着像，方向却相反。后者在同一 token 位置反复迭代，ECHO 是沿 horizon 往前走：每步以上一步的最后一层 hidden state 加一个轻量 fusion layer 为输入，递归调用主干最后一层和输出头。换句话说，**除了那层轻量 fusion layer，用于未来预测的每个参数，同时也是生成下一个 token 的参数**。\n\n## 深度 3 时推理 +2.63、代码 +2.23\n\n实验用 10B-A1B 的 latent MoE 骨干（24 层、256 个路由专家每 token 激活 8 个），预训练 600B token，在 192 张 H800 上跑约 3 天。评测覆盖 14 个 few-shot 基准，按数学、推理、多选、代码、闭卷问答五类统计。\n\n同一深度下对比 DeepSeek 的 MTP，深度 1 时 ECHO 在数学、推理、代码上分别领先 +0.79、+0.57、+1.35；深度 3 时推理拉到 +2.63，代码 +2.23。把每个方法自己从深度 1 拉到深度 3，ECHO 在 14 个基准上平均提升 **0.63 分**，DeepSeek 的 MTP 只有 **0.16 分**——这就是\"辅助模块饱和\"的量化证据。\n\n另两个数值得记：ECHO-1.5(3)（期望深度 1.5）拿到 ECHO-3 的同等收益，但 MTP 开销只有一半；ECHO-2(7)（期望深度 2）预训练阶段落后，SFT 后反超 0.49，对 DeepSeek MTP-Instruct 的领先达 +2.10。重复跑一遍预训练，ECHO 领先 +0.52 和 +0.78，推理与代码上的差距只波动 0.11——不是噪声。\n\n## 训练浅，投机深\n\n推理加速才是这篇论文真正的野心。共享同一层意味着训练好的模型可直接做草稿器，且**能超出训练时的深度去草稿**——辅助模块在深度外就饱和，ECHO 还在涨。它也彻底不需要事后微调草稿器：论文指出事后微调的草稿器训练数据比预训练语料小几个数量级，容易过拟合到学术基准。\n\n成本对比很直观：预训练约 3 天、SFT 5 小时；而事后草稿器微调用 24 张 H800，EAGLE-3 约 22 小时、DFlash 约 19 小时、DSpark 约 55 小时。ECHO 的摊薄 MTP 训练开销只有 **1.5 个 depth-1 步**。在按目标模型预测熵切分的开放式生成上，ECHO 保持草稿与目标的对齐，事后草稿器会丢掉这种对齐。\n\n## 我的判断\n\nECHO 值得注意的不是跑分，而是它把\"默认要多训一个模块\"的思路，换成了\"问主干最后一层能不能兼差\"。参数免费的代价是反复调用同一层，作者也承认深层监督噪声更大——ECHO-2(7) 预训练阶段反而落后，正是这个代价。\n\n论文明确列了限制：预训练只重复一次，微调只跑了一个 seed。GitHub 上已放出 ernie-research\u002FECHO 仓库。对正为草稿器微调发愁的团队，\"主干即草稿器\"值得先看。（来源：https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.24766）","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.24766","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"ad36daaa-5df8-47b7-a3f7-dce012819c94","en","ECHO: Baidu Puts Multi-Token Prediction in the Last Layer","A new arXiv paper from Baidu's ERNIE team argues that existing Multi-Token Prediction is auxiliary by design: the multi-horizon loss is absorbed by modules attached to the backbone, so the backbone never receives direct future-token supervision. ECHO instead recurses the backbone's own final decoder layer across every future horizon, and combines that with Dynamic Depth Sampling to hold amortized MTP overhead at 1.5 depth-1 steps. The shared layer also lets the pretrained model serve as its own speculative drafter, drafting past its trained depth while keeping draft-target alignment on open-ended, high-entropy generation.","Speculative decoding has become the default configuration for inference optimization: a small model drafts a few tokens, then a large model verifies them in one pass. But that setup carries an implicit prerequisite you may not have thought about — the drafter either has to be learned alongside training, or finetuned separately afterwards. What if neither is available? A paper that Baidu's ERNIE team posted to arXiv on October 17 argues that the backbone model can be its own drafter (arXiv:2610.24766).\n\n## Auxiliary modules absorb the supervision\n\nThe paper targets Multi-Token Prediction (MTP). Parallel MTP predicts several future positions independently from the same hidden state, ignoring causal dependencies among them. Causal MTP, as in DeepSeek-V3, restores the autoregressive dependency by feeding prediction modules the previous hidden state together with shifted token embeddings, and prediction quality improves markedly. Both, however, share one structural problem: future-token prediction is carried out by dedicated modules attached to the backbone, rather than being integrated into the autoregressive generation process itself.\n\nThere are two consequences. The first is indirect supervision: MTP losses primarily supervise the auxiliary prediction modules, so the backbone receives multi-horizon learning signals only indirectly. The second is that long horizons are expensive — each additional horizon adds a prediction module in memory and another pass in compute. That cost pins large-scale causal MTP pretraining to a single draft token: DeepSeek-V3 trains with one MTP module, and Xiaomi's MiMo found that adding MTP layers brings no further improvement.\n\nECHO sounds similar to looped transformers, but runs the opposite direction. Looped transformers iterate repeatedly at the same token position; ECHO moves forward along the horizon, feeding each step the previous step's final-layer hidden state through a lightweight fusion layer and recursively invoking the backbone's final decoder layer and output head. In other words, apart from that lightweight fusion layer, every parameter used for future prediction is also used for next-token generation.\n\n## The numbers: +2.63 on reasoning and +2.23 on code at depth 3\n\nExperiments use a 10B-A1B latent MoE backbone — 24 layers, 256 routed experts with eight active per token — pretrained on 600B tokens, roughly three days on 192 H800 GPUs. Evaluation covers fourteen few-shot benchmarks, grouped into mathematics, reasoning, multiple choice, code, and closed-book QA.\n\nCompared against DeepSeek's MTP at matched depth, ECHO leads by +0.79, +0.57, and +1.35 on mathematics, reasoning, and code at depth 1. At depth 3, the gap widens to +2.63 on reasoning and +2.23 on code. Taking each method from depth 1 to depth 3, ECHO gains 0.63 points averaged over the fourteen benchmarks while DeepSeek's MTP gains only 0.16 — that is the quantified evidence for auxiliary-module saturation.\n\nTwo more numbers are worth noting. ECHO-1.5(3), with an expected depth of 1.5, matches ECHO-3's gain at half the MTP overhead. ECHO-2(7), with an expected depth of 2, trails during pretraining but overtakes by 0.49 after SFT, reaching a lead over DeepSeek's MTP-Instruct of +2.10. On stability, repeating the pretraining run leaves ECHO ahead by +0.52 and +0.78, with the reasoning and code gap shifting by only 0.11. That is not noise.\n\n## Train shallow, draft deep\n\nThe real ambition of the paper is inference acceleration. Sharing one layer means the trained model can draft directly, and it can draft beyond the depth it was trained for — auxiliary modules saturate past their trained depth, while ECHO keeps gaining. It also removes any need for post-hoc drafter finetuning: the paper notes that such drafters are finetuned on a dataset orders of magnitude smaller than the pretraining corpus, which risks overfitting to academic benchmarks.\n\nThe cost comparison is concrete. Pretraining takes about three days, followed by five hours of SFT. Post-hoc drafter training uses 24 H800 GPUs: roughly 22 hours for EAGLE-3, 19 hours for DFlash, and 55 hours for DSpark. ECHO's amortized MTP training overhead is only 1.5 depth-1 steps. On open-ended generation, split by the target model's predictive entropy, ECHO keeps draft-target alignment where post-hoc drafters lose theirs.\n\n## My read\n\nWhat matters here is not the scores but the reframing. ECHO takes an assumption the field had grown comfortable with — that training a drafter needs an extra module — and asks instead whether the backbone's final layer can do double duty. The price for getting parameters for free is calling that layer repeatedly, and the authors acknowledge the supervision gets noisier at greater depth: ECHO-2(7) actually trailing during pretraining is that cost showing up in the data.\n\nThe paper is explicit about its limits. Pretraining was repeated once and finetuning ran with a single seed. The ernie-research\u002FECHO repository is already up on GitHub. For teams stuck paying to finetune a separate drafter, a backbone that drafts for itself is worth a look. (Source: https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.24766)","echo-baidu-multi-token-prediction-self-recursion","2026-10-21T01:15:00Z","2026-10-21T01:08:30.979233Z","2026-10-21T01:08:30.979251Z",true,"agent",90,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"368be0c3-aa88-42ae-a524-d0914f2088f7","PAR:循环模型多跑几轮就崩,两处训练改动修好外推","par-progress-aligned-recurrence-length-extrapolation","2026-10-21T11:10:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"ab1e6bdf-9433-41f0-be4d-cc0f78523ac2","SonicBoom v0.0.3:一个 .so 干掉手工调优","sonicboom-v003-mlir-compiler","2026-10-19T09:03:05+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"74cb8016-bdf7-4c8b-adbc-456d8c899b1a","EdgeWise 把 9B 模型训练成「喊人」高手:边云协同砍半成本","nvidia-edgewise-edge-cloud-slm-self-delegation-coding","2026-10-19T01:01:45+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"97ee186c-a42b-4389-8427-3342d77e8590","小模型何时喊云端大模型?NVIDIA 砍 62% 推理成本","edgewise-edge-agents-cloud-delegation","2026-10-16T21:07:57+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"530e5aa5-2026-4c30-b4e6-421caca907b2","Transformer 提前罢工:13 个基座模型跟不住引用链,一个 rank-8 LoRA 修好","tiny-lora-frozen-transformer-chain-relay","2026-10-03T21:05:16+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"9a77e0d7-8164-494e-984d-54cb57a7a0dd","Magnitude 开源:本地 Agent 专用推理引擎","magnitude-self-tuning-local-agent-inference-engine","2026-10-01T17:08:35+00:00"]