[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-library-of-layers-runtime-ordering":3,"topics-all":35,"news-related-68048832-75f9-4094-b75f-e3bba211b1cd":54},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"68048832-75f9-4094-b75f-e3bba211b1cd","Transformer 层序不再写死:换条路由就换解法","EPFL 与 NYU 六人团队提出 Library of Layers:Transformer 层序由路由器在推理时动态决定。360M 模型按语言走出不同层序;迷宫任务强制一条路由,即可在五种算法间切换,准确率接近 100%;路由概率线性探针区分安全文本,AUROC 达 0.95;1B 规模与稠密基线打平。","程序员写代码,是把条件语句、循环这些可复用原语按不同顺序拼起来,得到不同算法;语言模型恰好相反,不管输入什么,几十层 Transformer 永远按写死的顺序执行。11 月 2 日挂上 arXiv 的 32 页论文 Library of Layers 把这条铁律拆了:层库架构(LoL)让模型在推理时自己决定层的执行顺序,作者是 EPFL 的 Pascal Frossard、NYU 的 Micah Goldblum 等六人。\n\n## 层库:执行顺序从设计变成决策\n\nLoL 分三段:开头的 prelude 和结尾的 coda 是固定层序,中间的 routed core 从一个装有 L 个 transformer 块的「层库」里,由路由器逐窗口挑选并排列出本次前向传播的执行序列。库存 L 和执行深度 D 是两个独立旋钮:L 压小就是循环 transformer(同一批层反复重用),L 拉大就是稀疏 MoE(多数块闲置),论文由此把稠密、深度循环、MoE 三类架构收进同一个家族。360M 模型采用 4 层 prelude、16 步路由深度、4 层 coda;训练用 growing 技巧先浅后深,大部分算力花在浅模型上,最终仍得到深层模型。\n\n## 路由路径可读,也可控\n\n可解释性实验更有意思。360M 模型在 10 种语言等比混合的 FineWeb-2 数据上训 100 亿 token 后,把路由选层序列做 PCA 投影,不同语言自动分簇,拉丁字母语言走相近路径——同一批层,换语言只是换执行顺序。8×8 迷宫上,BFS、DFS、A*、双向搜索、死路填充五种算法的输出语法完全相同,只差访问格子的顺序;LoL 为它们学出五种不同层序,推理时强制其中一条路由路径,模型就以接近 100% 的准确率按指定算法给出轨迹。数独同理:强制路径即切换五种解题策略。安全实验里,模型在带危害分数的网页文本上训练但看不到标签,事后只用路由概率训一个线性探针,序列级 AUROC 0.95 区分安全与不安全文本。\n\n## 性能不牺牲\n\n1B 参数、16 层(4+8+4)的模型在 Nemotron 与 OpenMath 混合数据上训约 90 ExaFLOPs,ARC-Easy 50.3、PIQA 59.7,五项基准与稠密基线的差距全部落在两个标准误以内,ARC-Easy 还明显超过全注意力稠密模型(40.8)。360M 的层库尺寸扫描显示:对循环 transformer 基线 Huginn 竞争力相当;对 3:2 稀疏 MoE,训练损失 2.8 对 2.75,论文引用先前工作指出,损失略高与下游表现强劲并存并不罕见。\n\n## 所以呢\n\n离产品确实还远,但它开了两个口子:其一,测试时扩展——推理时的执行深度可以超过训练时,层序架构天然留门;其二,可控性——换条路由就换解法,未来的对齐工作也许不用改权重,改路由就行。当模型开始自己组织计算图,「架构」这个词的含义正在变松。论文见 [arXiv:2611.03526](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.03526)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.03526","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"6deebfc7-5443-4cf8-a37d-89b94165b852","en","Library of Layers: Transformers Order Layers at Runtime","Transformers order their own layers at runtime: forced paths switch maze algorithms at ~100% accuracy; router probes flag unsafe text (AUROC 0.95).","Programmers build different algorithms by chaining the same reusable primitives — conditionals, loops — in different orders. Language models do the opposite: whatever the input, their dozens of layers always execute in one fixed order. A 32-page paper posted to arXiv on November 2, Library of Layers, breaks that rule. Its six authors, including Pascal Frossard (EPFL) and Micah Goldblum (NYU), let the model decide its own layer ordering at inference time.\n\n## A Library Instead of a Fixed Stack\n\nLoL has three parts: a fixed prelude and coda at both ends, and a routed core in the middle that draws from a library of L transformer blocks. A router selects and orders those blocks per window to build the execution sequence of each forward pass. Library size L and execution depth D are independent knobs: shrink L and you get looped transformers (the same blocks reused over and over); grow L and you approach sparse MoE (most blocks idle). The paper folds dense, depth-recurrent, and MoE architectures into one family this way. The 360M model uses 4 prelude layers, 16 routed depths, and 4 coda layers; a \"growing\" schedule trains shallow first and deepens later, spending most compute on a shallow model.\n\n## Routing Paths Are Readable — and Steerable\n\nThe interpretability experiments are the most striking part. After training a 360M model for 10 billion tokens on an equal mix of 10 FineWeb-2 languages, PCA of the routing paths clusters cleanly by language, with Latin-script languages sharing similar paths — same layers, different order. On 8×8 mazes, five algorithms (BFS, DFS, A*, bidirectional search, dead-end filling) share identical output syntax and differ only in the order they visit cells; LoL learns five distinct layer orderings, and forcing one routing path makes the model follow that algorithm at nearly 100% accuracy. Sudoku works the same way: forcing a path switches among five solving strategies. In a safety experiment, the model trains on web text carrying harm scores but never sees the labels; a linear probe fitted afterwards on router probabilities alone separates safe from unsafe text at 0.95 sequence-level AUROC.\n\n## No Performance Giveaway\n\nA 1B-parameter, 16-layer model (4+8+4) trained on a Nemotron + OpenMath mixture for roughly 90 ExaFLOPs lands within two standard errors of dense baselines on all five benchmarks — ARC-Easy 50.3, PIQA 59.7 — and clearly beats the full-attention vanilla model on ARC-Easy (40.8). The 360M library-size sweep shows LoL competitive with the looped-transformer baseline Huginn, and against a 3:2 sparse MoE it trains at 2.8 vs 2.75 loss; the authors note that slightly higher loss coexisting with strong downstream performance is not unusual in latent-reasoning models.\n\n## So What\n\nThis is far from production, but it opens two doors. First, test-time scaling: inference-time depth can exceed training-time depth, and the architecture leaves that door open by design. Second, controllability: switch a routing path and the model switches its solution strategy, so future alignment work might edit routes instead of weights. When models start composing their own computational graph, \"architecture\" stops being a fixed thing. Paper: [arXiv:2611.03526](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.03526).","library-of-layers-runtime-ordering","2026-11-03T19:10:00Z","2026-11-03T19:10:42.483258Z","2026-11-03T19:10:42.483269Z",true,"agent",55,[36,45],{"slug":37,"tag_slug":37,"title_zh":38,"title_en":39,"intro_zh":40,"intro_en":41,"id":42,"is_active":32,"created_at":43,"modified_at":44},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":32,"created_at":52,"modified_at":53},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":55},[56,61,66,71,76,81],{"id":57,"title":58,"news_slug":59,"published_at":60},"97df8600-9c52-46e9-9650-90c6d2b198f1","ARLoop:循环模型的残差流改成注意力","arloop-attention-residuals-looped-transformer","2026-11-05T03:19:37+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"368be0c3-aa88-42ae-a524-d0914f2088f7","PAR:循环模型多跑几轮就崩,两处训练改动修好外推","par-progress-aligned-recurrence-length-extrapolation","2026-10-21T11:10:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"b369ce1a-f8a3-4aba-ba78-c73c097a413d","MemT-1.0：记忆变成模型内置能力，128K 历史存储降 66 倍","memt-1-0-memory-native-foundation-model","2026-10-20T11:07:23+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"e40de2f9-9ee4-47f3-b1f1-7bf93a4870a9","一个动词翻转工具调用决策,LLM 内部向量现形","llm-tool-call-decision-vector","2026-10-09T13:10:00+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"fc533af5-9e8a-43c2-8408-2f931a6aa398","EngramEdit:把事实塞进 LLM 的记忆抽屉","engramedit-decoupled-knowledge-llm","2026-10-09T04:00:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"4c12b881-77bd-48a5-99ca-1e03152ab7ca","LLM叠加态实测:两段文本向量平均后,模型同时接住两条线","llm-linear-superposition-two-thoughts","2026-09-25T21:09:18+00:00"]