[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-nope-distance-generalization-transformers":3,"topics-all":35,"news-related-c9fa6b72-900c-4064-b7a4-fe4b517dc740":54},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"c9fa6b72-900c-4064-b7a4-fe4b517dc740","RoPE 白加了?关掉位置编码,Transformer 距离泛化反而更强","arXiv 新论文提出「距离泛化」测法:上下文长度不变,只拉开 token 间隔。合成拷贝任务实测,不加位置编码的 NoPE 在多数场景泛化最好,RoPE 反而垫底;数据多样性收益递减,任务迁移可正可负。","RoPE 如今是大语言模型的业界标配,Llama、DeepSeek 等主流模型都在用,但它的长度外推能力一直被诟病,位置插值、YaRN 等补丁层出不穷。9 月 10 日,法兰克福歌德大学的 Nevermann 与 Gros 在 [arXiv:2609.11913](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.11913) 上提出一个更基础的问题:位置编码对「距离泛化」到底有没有帮助?实测答案有点反直觉——多数情况下,把它整个关掉反而最好。\n\n## 先分清两种泛化\n\n长度泛化研究「训练短上下文、推理长上下文」,模型同时面对没见过的位置和没见过的依赖关系,两种失败混在一起。这篇论文定义距离泛化:上下文长度固定,只改变源 token 与复述位置之间的间隔,把「处理不了新位置」和「处理不了新依赖」拆开看。实验用两个合成延迟拷贝任务:完整拷贝要延迟复述 10 个源 token,选择性拷贝只复述其中的偶数 token。训练时间隔在 15 到 25 之间均匀采样,测试时间隔拉到 1-120,大量超出训练分布。\n\n## NoPE 多数场景拿下最佳\n\n模型是 8 层、8 头、512 维的标准 decoder-only Transformer,分别跑 RoPE、ALiBi 和不加位置编码的 NoPE 三种配置。结果显示距离泛化对位置编码方案高度敏感,NoPE 在多数设置下泛化最好,与 Kazemnejad 等人 2023 年在长度泛化上的结论一致。更扎心的是 RoPE:它在距离泛化上落后,而测试序列并没有超出训练上下文,RoPE 并未遇到未训练过的旋转角度——锅甩不到「外推到陌生角度」头上,机制层面另有原因。\n\n## 数据多样性与迁移的两个附带发现\n\n第二组结论关于训练数据多样性:加大训练时见过的间隔数量,绝对性能确实提升,但按相对口径衡量,收益强烈递减。第三组关于迁移学习:主任务与辅助任务同训,收益可正可负——从更复杂的选择性拷贝迁往完整拷贝出现正向收益,但在完整拷贝自己的训练区间内反而出现破坏性效应;RoPE 的迁移整体偏弱,却有一个独特优点:几乎不产生破坏性干扰。\n\n## 先别急着拆 RoPE\n\n两个限定值得记住。其一,NoPE 对模型规模敏感:模型变小时,NoPE 甚至学不会分布内的拷贝机制,显式位置编码在小模型上稳定得多。其二,这是 8 层小模型加合成任务的结论,作者在讨论里写明,距离泛化最终需要拿到预训练模型上检验,从合成任务到 LLM 之间还隔着规模、预训练数据和多任务混杂好几道沟。\n\n对做长上下文工程的人来说,这篇论文的价值不在「立刻换掉 RoPE」,而在把一个被混淆的变量拆干净了:很多「长上下文不行」的问题,可能不是位置编码外推的锅,而是模型没学会处理没见过的 token 依赖。下次 RULER 分数上不去时,或许该先问:到底是位置没见过,还是依赖没见过?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.11913","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"e2518149-a480-4ee0-b9ca-72e234e684a2","en","Transformers may generalize better without positional encoding","arXiv paper tests distance generalization at fixed context: NoPE beats RoPE and ALiBi on synthetic tasks; diversity gains diminish; transfer cuts both ways.","RoPE is the industry standard for large language models, employed by major families like Llama and DeepSeek, yet its length extrapolation has long been criticized, spawning patches such as positional interpolation and YaRN. On September 10, Daniel Henrik Nevermann and Claudius Gros of Goethe University Frankfurt posted a paper ([arXiv:2609.11913](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.11913)) asking a more basic question: does positional encoding actually help with \"distance generalization\"? The measured answer is counterintuitive — in most cases, turning it off entirely works best.\n\n## Two kinds of generalization\n\nLength generalization studies models trained on short contexts and evaluated on longer ones, so unseen positions and unseen token dependencies fail together, hopelessly confounded. This paper defines distance generalization: context length stays fixed while the gap between source tokens and their recall varies, separating \"can't handle unseen positions\" from \"can't handle unseen dependencies\". Two synthetic delay-copy tasks are used: full copy recalls 10 source tokens after a delay, while selective copy recalls only the even-numbered tokens. Training distances are sampled uniformly between 15 and 25; test distances stretch from 1 to 120, far outside the training distribution.\n\n## NoPE wins in most settings\n\nThe model is a standard 8-layer, 8-head, 512-dimension decoder-only Transformer, run in three configurations: RoPE, ALiBi, and NoPE (no positional encoding). Distance generalization turns out to be highly sensitive to the encoding scheme, with NoPE generalizing best in most settings — consistent with Kazemnejad et al.'s 2023 findings on length generalization. The sting for RoPE: it falls behind even though test sequences never exceed the training context, so RoPE never faces untrained angular regimes. The blame cannot be pinned on extrapolating to unfamiliar angles; the mechanism lies elsewhere.\n\n## Two side findings: diversity and transfer\n\nOn data diversity, enlarging the set of inter-token distances seen in training lifts absolute performance, but relative returns diminish strongly. On transfer learning, training a main task together with an auxiliary one cuts both ways: transferring from the more complex selective copy to full copy yields positive gains, yet destructive effects appear within full copy's own training range. RoPE transfers weakly overall but has one distinct advantage: almost no destructive interference.\n\n## Don't rip out RoPE just yet\n\nTwo caveats matter. First, NoPE is sensitive to model scale: as models shrink, NoPE even fails to learn the in-distribution copy mechanism, while explicit encodings remain far more stable. Second, these are results from an 8-layer model on synthetic tasks; the authors state in their discussion that distance generalization ultimately needs to be tested on pretrained models — scale, pretraining data, and multi-task mixing still stand between synthetic tasks and LLMs.\n\nFor long-context engineers, the value of this paper is not \"drop RoPE now\" but a clean unconfounding of variables: many \"long context fails\" problems may not be positional extrapolation's fault at all, but the model never learning to handle unseen token dependencies. Next time RULER scores stall, ask first: is it the positions that are unseen, or the dependencies?","nope-distance-generalization-transformers","2026-09-11T13:10:00Z","2026-09-11T13:11:05.602463Z","2026-09-11T13:11:05.602481Z",true,"agent",59,[36,45],{"slug":37,"tag_slug":37,"title_zh":38,"title_en":39,"intro_zh":40,"intro_en":41,"id":42,"is_active":32,"created_at":43,"modified_at":44},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":32,"created_at":52,"modified_at":53},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":55},[56,61,66,71,76,81],{"id":57,"title":58,"news_slug":59,"published_at":60},"2266cea6-06f1-4932-8905-1bc3f2e5a8c0","Meta FAIR 字节蒸馏研究:End-Of-Token 渐近反超 token 蒸馏 4%,数据只需 1\u002F6","meta-fair-byte-distillation-token-ceiling-2026-09","2026-09-15T02:00:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"b4f270b3-db43-4586-a0e5-a062320c6d1b","让模型自己声明看哪里:Declarative Attention 零训练砍 52% KV 读取","declarative-attention-kv-cache-declare","2026-09-03T23:07:03+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"b5638cab-a4d6-44ac-9230-32ed0a4cba9d","ARMT 把「记忆」焊进 Transformer:用恒定显存换无限上下文","armt-associative-recurrent-memory-transformer","2026-07-23T00:00:00+00:00"]