[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-arloop-attention-residuals-looped-transformer":3,"topics-all":38,"news-related-97df8600-9c52-46e9-9650-90c6d2b198f1":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"97df8600-9c52-46e9-9650-90c6d2b198f1","ARLoop:循环模型的残差流改成注意力","arXiv 新论文 ARLoop 把注意力残差搬进循环深度:每层沿深度用注意力挑选要复用的历史表示。受控实验中困惑度优于既有残差方案;放大到 1.2B、0.5T token 后,论文表格显示全项超过 3.5B 的 Huginn,并与规模相近的 Qwen3.5-2B 打平或更好。训练代码与中间检查点全部开放。","Transformer 有一个部件,我认为至今仍是整条流水线里改动最少的:残差连接。它把每层输出加回主干,粗暴但稳定,绝大多数研究都在它周围打转——加门控、加缩放、加多条车道。直到最近,才有人认真去动它。俄亥俄州立大学 NLP 组(OSU NLP Group)放出的 ARLoop 把一个更激进的思路搬进了循环模型:让残差流自己决定该复用哪几层。\n\n结果有点反常识:一个 1.2B 参数、只训了 0.5T token 的循环模型,在[论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.05234)的评测表里全面超过了 3.5B 参数、训练时最多用到 64 次循环的 Huginn,论文称它还与训练 36T token 的 Qwen3.5-2B 打平或更好。训练代码、数据处理脚本、中间 checkpoint 全部开放。\n\n## 残差流:一条没有选择机制的管道\n\n标准 PreNorm Transformer 里,第 i 层的输入是 h_i = h_{i-1} + f_i(Norm(h_{i-1}))。把它展开就能看到一个事实:任意更早一层产生的残差更新,都以固定的单位系数一路加到后面所有层。这条单一残差流没有\"挑选\"机制——后面的层想用哪一层的输出,只能用或者不用,不能选。\n\nARLoop 借的是 Kimi 团队 2026 年提出的注意力残差(Attention Residuals):把\"深度\"本身当成一个可以被 attention 的维度。每一层用自己学到的查询向量,对前面若干个残差源做 softmax,自己决定聚合权重。为控制开销,论文采用块级形式——把连续若干子层的残差更新合成一个块级表示,块内照旧相加,块与块之间走注意力。\n\n## 循环深度下,历史其实是冗余的\n\n论文里有个反直觉的发现:在循环模型上,只让深度注意力看最近几个残差块(外加输入嵌入),效果不但不差,在更大 R 下还更好——而同样的滑窗限制搬到标准 Transformer 上是会削弱收益的。\n\n论文用注意力可视化与余弦相似度解释了原因:在 140M 规模的 ARLoopfull、R=4 的可视化里,第 47 个有效子层把 96% 的注意力权重交给了最近三个残差块加嵌入块。根源是参数共享——循环反复套用同一批层,不同轮次产出的表示高度相似,远端历史提供不了多少新信息,反而可能引入干扰。\n\n## 放到真实预训练配方里\n\n受控实验(140M 与 370M 两档,FineWeb-Edu 语料 11.2B 与 29.6B token,上下文 2048)里,ARLoop 在各档 R 上困惑度都最低:140M 的 R=3 是 16.38,PreNormLoop 16.91、Parcae 16.60(带输入注入 16.55)、DeepLoop 17.05、Sandwich-Norm 17.19;370M 的 R=3 是 12.17,PreNormLoop 12.48。140M 档上相对 PreNormLoop 的增益从 R=1 的 0.11 涨到 R=3 的 0.53,说明它确实更会用循环深度换来的那份额外算力。\n\n放大到真实配方(SmolLM 风格,GQA 加 SwiGLU,数据用 FineWeb-Edu、FineMath 与 Stack-Edu,上下文 4096)之后,0.6B 模型在 180B token 上对比同架构、同预算的 PreNormLoop-0.6B:GSM8K 40.9 对 35.7,MATH 26.3 对 20.9,MBPP 36.2 对 33.1,八项评测里赢七项;唯一输掉的是 ARC-C,45.5 对 46.3。1.2B 模型(24 层、R=3、500B token)的表更激进:GSM8K 63.1、MATH 37.8、MBPP 52.1、HumanEval 33.1。论文称它在全部评测任务上超过 Huginn-3.5B(后者 0.8T token 档的 GSM8K 是 26.1、MATH 14.1),并称与 Qwen3.5-2B 在所评基准上打平或更好(Qwen3.5-2B 的 GSM8K 是 60.6、MATH 35.8)。\n\n## 三笔要算的账\n\n第一笔是吞吐。ARLoopfull 比 PreNorm 平均慢约 15%,换成滑窗版本把差距压到约 10%;内存几乎不变(0.6B 档 66.5 GiB 对 66.6 GiB)。论文说深度注意力增加的算术量很少,主要开销是额外的显存带宽访问,后续 kernel 优化还有压缩空间。\n\n第二笔是数据。论文自己承认 ARLoop-1.2B 在多数任务上不如 Ouro-1.4B,后者 GSM8K 78.2、MATH 70.5,归因是对方预训练数据预算更大、数据源质量更高。1.2B 用 0.5T token 追平 36T token 的 Qwen3.5-2B 看着漂亮,但这更多说明基座小不等于账算得划算——换成规模相近的非循环模型,数据预算是躲不掉的。\n\n第三笔是外推。循环模型的一个卖点是训练时 R=3、推理时多跑几轮。ARLoop 的结论偏冷静:性能涨到训练时最大 R 为止,再往上不涨。也就是说,把残差访问改聪明了,不会自动解锁推理期的循环外推——\"在训练深度内组织好信息\"与\"外推到没训过的深度\"是两个问题。\n\n## 完全开放意味着什么\n\n[代码库](https:\u002F\u002Fgithub.com\u002FOSU-NLP-Group\u002FARLoop)放出了数据准备、模型实现、训练配方和评测代码;[模型卡](https:\u002F\u002Fhuggingface.co\u002Fosunlp\u002FARLoop-1.2B)放出了 1.2B 这条 500B token 训练线的全程 checkpoint——第 1 万步到第 23 万步共 23 个中间检查点,加上第 238,417 步的最终版本,总训练量 499,996,688,384 token,Apache 2.0 授权。结构上,24 个物理层共享着跑 3 轮,有效深度 72,隐藏维 2048,MLP 中间维 5632,16 个注意力头配 4 个 KV 头,上下文 4096。\n\nLoop 方向这两年出了不少论文,但大多只放权重,或者干脆不放。这篇把训练轨迹完整摊开,意味着别人可以从中段 checkpoint 接手,不必从零复现一遍训练——按论文自己的账,受控实验总共烧掉约 12K GPU 小时,0.6B 用 8 张 H100 跑了约一周,1.2B 用 16 张 B200 跑了约两周。\n\n## 所以呢\n\n残差流这根管道被撬开之前,大多数人的注意力都在层数、宽度和激活函数上。ARLoop 提醒的是另一件事:模型的深度不只是物理层数,还有信息在层之间流动的拓扑。循环模型让同一批层反复使用,冗余必然产生,这时候让模型自己挑历史,比让它无差别全接更划算。\n\n不过这篇更像一份可行性证明,而不是结论:1.2B、0.5T token 的体量,还谈不上下任何关于循环路线的判断。真正值得等的是它能不能在更大规模上稳住那 10% 到 15% 的吞吐代价——如果能,残差流改造就不再是实验室技巧,而是每个循环模型都该抄的一笔。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.05234","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"56b30d4f-fcce-43af-9f36-709dcb52fb3e","en","ARLoop: Attention Residuals in a 1.2B Looped Transformer","ARLoop ports attention residuals into recurrent depth; at 1.2B, the paper’s tables report it beating Huginn-3.5B on every evaluated task.","There is one component of the Transformer that I think has gone largely untouched for the better part of a decade: the residual connection. Every layer’s output gets added back into the trunk. It is blunt, it is stable, and nearly all subsequent work has circled around it rather than through it — gating it, scaling it, splitting it into multiple lanes. Until recently, somebody finally went after it properly. The ARLoop work released by the NLP group at Ohio State takes a more radical idea and carries it into looped transformers: let the residual stream decide for itself which earlier layers are worth reusing.\n\nThe result is somewhat counterintuitive. A 1.2B-parameter looped model trained on just 0.5T tokens is reported to beat Huginn, a 3.5B-parameter model trained with up to 64 recurrences, across every benchmark in the [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.05234)’s evaluation table, and, the paper claims, matches or exceeds Qwen3.5-2B, a non-looped model trained on 36T tokens. The training code, the data-processing scripts, and the intermediate checkpoints are all released.\n\n## The residual stream as a pipe with no choices\n\nIn a standard PreNorm Transformer, the input to layer i is h_i = h_{i-1} + f_i(Norm(h_{i-1})). Unroll that and one fact becomes obvious: every residual update produced by an earlier layer gets added, with a fixed unit coefficient, into every residual state that comes after it. This single residual stream offers no mechanism for selection. A later layer that wants to use an earlier layer’s representation can either take the whole stream or not; it cannot pick and choose.\n\nARLoop borrows from the attention residuals proposed by the Kimi team in 2026, which treat depth itself as something attention can operate over. Each layer uses its own learned query vector to run a softmax over the available residual sources and decide the aggregation weights on its own. To keep the overhead manageable, the paper uses a block-level form: consecutive sublayers are summed into a block-level residual representation, addition happens inside a block as in PreNorm, and attention operates across blocks.\n\n## Under recurrent depth, history turns out to be redundant\n\nThe paper contains a counterintuitive finding. On looped models, letting depth attention see only the most recent residual blocks, plus the input embedding, is not merely competitive — at larger recurrence depths it is better. The same sliding-window restriction applied to a standard transformer reduces the gains, so the contrast is doing real work here.\n\nAttention visualizations and pairwise cosine similarity explain why. In the 140M-scale ARLoopfull at R=4, sublayer 47 places 96% of its total attention weight on the three most recent residual blocks plus the embedding block. The culprit is parameter sharing. Recurrence applies the same layers over and over, so representations produced at different recurrence depths end up highly similar. Distant history carries little new information and can even introduce interference.\n\n## Moving to a realistic pretraining recipe\n\nIn the controlled experiments, at 140M and 370M parameters, trained on 11.2B and 29.6B tokens of FineWeb-Edu at a context length of 2048, ARLoop posts the lowest perplexity at every recurrence depth tested. At 140M with R=3 it reaches 16.38, against 16.91 for PreNormLoop, 16.60 for Parcae without input injection and 16.55 with it, 17.05 for DeepLoop, and 17.19 for Sandwich-Norm. At 370M with R=3 it reaches 12.17 against 12.48 for PreNormLoop. At the 140M scale, the gain over PreNormLoop grows from 0.11 at R=1 to 0.53 at R=3, which suggests the method genuinely exploits the extra computation that recurrent depth buys.\n\nScale up to a realistic recipe — SmolLM-style, grouped-query attention with SwiGLU, FineWeb-Edu, FineMath and Stack-Edu as data, context length 4096 — and the picture sharpens. The 0.6B model, trained on 180B tokens, is compared against an architecturally identical PreNormLoop-0.6B on the same budget: GSM8K 40.9 against 35.7, MATH 26.3 against 20.9, MBPP 36.2 against 33.1. It wins seven of eight evaluated tasks. The one it loses is ARC-C, 45.5 against 46.3. The 1.2B model, with 24 layers, R=3 and 500B tokens, is more aggressive still: GSM8K 63.1, MATH 37.8, MBPP 52.1, HumanEval 33.1. The paper claims it outperforms Huginn-3.5B on every evaluated task — that model’s 0.8T-token run scores 26.1 on GSM8K and 14.1 on MATH — and claims it matches or exceeds Qwen3.5-2B on the evaluated benchmarks, where Qwen3.5-2B scores 60.6 on GSM8K and 35.8 on MATH.\n\n## Three bills worth adding up\n\nThe first is throughput. ARLoopfull runs about 15% slower than PreNorm on average; the sliding-window variant cuts that gap to roughly 10%. Memory is essentially unchanged — 66.5 GiB against 66.6 GiB at the 0.6B scale. The paper argues depth attention adds little arithmetic and that the main cost is extra memory bandwidth traffic, which leaves room for kernel optimization later.\n\nThe second is data. The paper itself concedes that ARLoop-1.2B trails Ouro-1.4B on most tasks — Ouro scores 78.2 on GSM8K and 70.5 on MATH — and attributes the gap to a substantially larger pretraining data budget and higher-quality data sources. Matching a 36T-token Qwen3.5-2B with 0.5T tokens looks impressive, but it mostly says that a small base model does not mean a small bill. At the same parameter count, a non-looped model still has to pay the data budget.\n\nThe third is extrapolation. One of the selling points of looped models is training at R=3 and running more recurrences at inference. ARLoop’s conclusion here is cool to the point of deflationary: performance improves up to the maximum training recurrence and does not improve beyond it. Making residual access smarter does not automatically unlock inference-time recurrence extrapolation. Organizing information within the trained depth range and generalizing recurrent computation past that range turn out to be separate problems.\n\n## What full openness actually buys\n\nThe [codebase](https:\u002F\u002Fgithub.com\u002FOSU-NLP-Group\u002FARLoop) ships the data preparation pipeline, model implementation, training recipes and evaluation code. The [model card](https:\u002F\u002Fhuggingface.co\u002Fosunlp\u002FARLoop-1.2B) ships the full checkpoint trail for the 1.2B 500B-token run: 23 intermediate checkpoints from step 10,000 through step 230,000, plus the final revision at step 238,417, for 499,996,688,384 training tokens total, under an Apache 2.0 license. Architecturally, 24 physical layers are shared across 3 recurrent passes for an effective depth of 72, with hidden width 2048, MLP intermediate width 5632, 16 attention heads against 4 KV heads, and a context length of 4096.\n\nPlenty of papers have come out of the loop direction over the past two years, but most release weights only, or nothing at all. This one lays the entire training trajectory out, which means someone else can pick up from a mid-run checkpoint instead of reproducing a training run from scratch. The paper’s own accounting: roughly 12K GPU hours for the controlled experiments, about a week on 8 H100s for the 0.6B model, about two weeks on 16 B200s for the 1.2B one.\n\n## So what\n\nBefore the residual stream was pried open, most attention went to layer counts, hidden widths and activation functions. ARLoop points at something else: a model’s depth is not just its physical layer count, it also includes the topology through which information flows between layers. Looped models reuse the same batch of layers, so redundancy is unavoidable, and in that setting letting the model pick its own history beats making it accept everything indiscriminately.\n\nStill, this reads more like a proof of feasibility than a conclusion. A 1.2B model trained on 0.5T tokens is not enough to settle any judgement about the loop direction. What is worth waiting for is whether the 10% to 15% throughput cost holds steady at larger scale. If it does, the residual-stream rewrite stops being a laboratory trick and becomes something every looped model should copy.","arloop-attention-residuals-looped-transformer","2026-11-05T03:19:37Z","2026-11-05T03:19:39.854681Z","2026-11-05T03:19:39.854688Z",true,"agent",4,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"e40de2f9-9ee4-47f3-b1f1-7bf93a4870a9","一个动词翻转工具调用决策,LLM 内部向量现形","llm-tool-call-decision-vector","2026-10-09T13:10:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"68048832-75f9-4094-b75f-e3bba211b1cd","Transformer 层序不再写死:换条路由就换解法","library-of-layers-runtime-ordering","2026-11-03T19:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"022731b7-2aa0-435f-b93e-3f84eea2699a","R2Orch双层自博弈:技能内化,零技能上阵","r2orch-self-play-skill-withdrawal","2026-11-02T21:10:07+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"eb55e1cf-15c1-487f-845c-e81f87f11cfa","红白机 CPU 编进 Qwen 权重:没训练,纯编译","nesformer-qwen-nes-cpu-weights","2026-10-31T17:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"41889a7a-a82d-4e8e-b693-7db85a4f62c7","CMU统一训练:数学分打平,下一轮代码反超分阶段","unified-training-single-likelihood-foundation-models","2026-10-31T09:16:57+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"be62f4f6-6935-4c2c-abe8-9e0732bb04ef","Trajectory2Gym:智能体的成功轨迹不等于好教材","trajectory2gym-compositional-interaction-synthesis","2026-10-27T21:10:28+00:00"]