[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-loopmtp-latent-multi-token-loop-guidance":3,"news-related-e3f049f5-2f0d-48d2-8e88-246ef006fa16":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","循环 Transformer 通过重复使用同一组层换取有效深度，却容易让不同迭代变成重复劳动，甚至把早期正确判断改坏。LoopMTP 将每轮隐藏状态与更远未来 token 的嵌入软对齐，再用内容条件门控汇总各轮表示；论文报告其在固定参数预算下获得最高 8.1% 的相对平均准确率提升，并展示了循环次数增加后的稳定性与明确边界。","\n循环 Transformer 有一个很诱人的思路：不再不断堆叠新的独立层，而是让同一组 Transformer 层重复运行若干次，用固定参数量换取更深的有效计算。问题在于，**循环不等于进步**。研究团队指出，现有循环模型会覆盖中间表示，导致早期已经正确的预测在后续迭代中被改坏；不同轮次的隐藏表示还会越来越相似，让新增计算沦为重复劳动。\n\nLoopMTP 的核心，就是给每一轮循环安排一个不同的“前瞻目标”。对于第 t 轮，模型不直接做一次昂贵的全词表预测，而是在隐藏空间里，让当前位置的表示与第 t 个未来 token 的输出嵌入做余弦软对齐。第 1 轮保持无约束，用来学习可供后续轮次读取的丰富表示；从第 2 轮开始，每一轮都被要求朝更远的未来移动。论文称，这种潜空间多 token 预测不需要每轮增加一次完整词表投影，因此额外开销很小。具体方法与实验可在[论文原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03624)中追溯。\n\n光有前瞻监督还不够。传统循环结构常常只保留最后一轮输出，前面算出的信息会被覆盖。LoopMTP 改用内容条件门控，把所有轮次的表示加权汇总；同时加入轮次索引、逐轮归一化，以及固定按循环次数缩放的 Loop-LNS，让共享骨干知道自己正处在哪一轮，并控制深层展开时的残差稳定性。换句话说，它不是简单地把同一层多跑几遍，而是同时解决“每轮做什么”和“前面算过什么如何保留”两个问题。\n\n论文报告，主实验结果按 3 个随机种子取平均。LoopMTP 在通用任务上的平均准确率最高达到 50.02%，非循环参数匹配基线为 46.28%，相对提升最高 8.08%；与 LoopFormer 在相同循环次数下比较时，28 组匹配结果中有 27 组更好。更关键的是，困惑度与通用任务准确率会随循环次数增加而持续改善，但问答、数学和代码任务在 7 轮后趋于饱和，并在 9 轮时略有回落。这说明更多循环不是免费午餐，真正有用的是受到区分性监督的循环。\n\n研究者还训练了一个约 2.6 亿参数的数学专用模型，使用约 68 亿个数学语料 token，未做微调或指令微调。在 GSM8K 的 8-shot 设置下，它取得 19.03% 准确率，而参数量匹配的非循环基线为 7.05%。不过，这组专用模型实验只覆盖数学领域，而且每个配置只运行了一个随机种子；论文也明确承认，循环增益会递减甚至出现非单调变化，实用循环深度的上限仍未确定。\n\n我的判断是，LoopMTP 的价值不在于又刷出一个孤立分数，而在于指出了循环模型最容易被忽略的工程事实：**参数共享只能省下存储，每轮监督才能让重复计算产生增量**。它为小模型和本地部署提供了一条清晰路线——用更多可控计算换取更强推理，同时不扩大参数内存；但在更大规模、更多领域验证完成之前，还不能把小模型结果直接外推到前沿模型。\n\n所以，循环 Transformer 的下一步不是“多想几轮”，而是让每一轮都知道自己为什么要想。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03624","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"ddb12c01-039f-4404-ae9b-d15ea249eead","en","LoopMTP gives looped Transformers a look-ahead roadmap","Looped transformers reuse the same layers to gain effective depth, but their iterations can become redundant or even overwrite an earlier correct prediction. LoopMTP softly aligns each loop's hidden state with a farther future token embedding and aggregates all iterations through a content-conditioned gate. The paper reports up to an 8.1% relative average-accuracy gain at a fixed parameter budget while also showing where additional loops stop helping.","# LoopMTP: Giving Every Transformer Loop a Direction\n\nLooped transformers offer an appealing bargain: instead of stacking more independent layers, they run the same Transformer stack several times, trading additional computation for greater effective depth without increasing the unique parameter count. The problem is that **repetition is not automatically progress**. The LoopMTP paper argues that current looped models overwrite intermediate representations, so a prediction that is correct after an early pass can be revised into an error. Their hidden states can also become increasingly similar across iterations, turning additional compute into redundant work.\n\nLoopMTP addresses this by assigning a distinct lookahead target to each pass. At loop t, the model does not perform another expensive full-vocabulary prediction. Instead, it softly aligns the current hidden representation with the output embedding of the token t positions in the future, using cosine similarity in latent space. The first loop remains unconstrained so that it can learn a rich representation for later passes; from the second loop onward, each iteration is guided toward a progressively more distant future token. According to the paper, this latent multi-token prediction signal avoids a separate vocabulary projection at every loop and therefore adds negligible overhead. The method and results are documented in the [original paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03624).\n\nLookahead supervision alone is not enough. Conventional looped architectures often keep only the final pass, discarding useful information computed earlier. LoopMTP instead combines all iteration states through a content-conditioned gate. It also supplies an iteration index, uses iteration-specific normalization, and introduces Loop-LNS, which applies a fixed scaling factor based on the number of loops to stabilize the residual stream during deep unrolling. In practical terms, the design tackles two separate questions: what should each loop compute, and how should useful information from earlier loops survive?\n\nThe main experimental results are averaged over three random seeds. On general tasks, LoopMTP reaches an average accuracy of 50.02%, compared with 46.28% for a parameter-matched non-looped baseline, corresponding to a maximum relative improvement of 8.08%. Against LoopFormer at matched loop counts, it performs better in 27 of 28 comparisons. Perplexity and general-task accuracy improve as the loop count rises, but the QA, mathematics, and code suites improve through seven loops and then regress slightly at nine. That boundary matters: more iterations are not a free source of intelligence. What helps is differentiated, supervised computation.\n\nThe researchers also train a mathematics-specialized model with roughly 260 million parameters on about 6.8 billion tokens of mathematical data, without fine-tuning or instruction tuning. Under an 8-shot GSM8K evaluation, it reaches 19.03% accuracy, while the parameter-matched non-looped baseline reaches 7.05%. The paper is careful about the scope of this result: the specialist experiment covers only mathematics, each specialist configuration uses a single random seed, and gains can diminish or become non-monotonic as the number of loops grows. The useful upper limit of recurrent depth remains unresolved.\n\nMy reading is that LoopMTP matters less as an isolated benchmark result than as an engineering lesson for recurrent model design: **parameter sharing saves memory, but per-loop supervision is what makes repeated computation additive**. That makes it relevant to small models and on-premises deployment, where additional controlled compute may be easier to provide than additional parameter memory. But the small-model findings should not be extrapolated to frontier-scale systems until the architecture is tested at larger scale and across more domains.\n\nThe next step for looped transformers is not simply to “think for more rounds.” It is to make every round know why it is thinking.","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09Z","2026-08-06T00:23:47.446246Z","2026-08-06T00:23:47.446256Z",true,"agent",125,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"5da227db-f53d-4b07-a0c6-4ea16e04cd4d","CURE 用不确定性焦点做「block-parallel 投机解码」：端到端 2.66–3.49×、接受长度涨 4.2–7.5%","cure-block-parallel-speculative-decoding","2026-08-08T02:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"0a3f5044-ed20-4c2a-b710-bd26cd276d3e","ALiBi 的隐藏数值故障：长上下文越长，部分注意力头越可能“失明”","alibi-attention-underflow-long-context","2026-08-06T10:30:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"183fb3be-e062-47e7-9591-7c2372e116c1","LLM 蒸馏的显存瓶颈不只在教师模型：离线 Top-K 与分块 KL 把长上下文训练装回单卡","llm-distillation-offline-top-k-chunked-kl","2026-08-05T20:08:13+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"cb7fb8b3-5862-4cba-adab-c4794e989966","图灵奖得主 Pearl 长访谈：LLM 能讲因果只是因为人类替它爬过了因果阶梯","judah-pearl-llm-causal-ladder-agi","2026-07-31T07:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"c48681ff-ddbb-402c-ade8-23b584a06aea","更强教师反而教不动学生：Lightning OPD 2.0 剥掉蒸馏中的“文风噪声”","lightning-opd-2-cross-teacher-style-bias","2026-07-30T16:17:15+00:00"]