[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-taylor-calibrate-transformer-gdn-distill-88x":3,"news-related-f63a58a9-85c9-406c-beef-0ba1cb0c6985":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"f63a58a9-85c9-406c-beef-0ba1cb0c6985","Taylor-Calibrate 把 Transformer 蒸馏成 GDN 的初始化做成系统级工程","当下把 Transformer 转成 Gated DeltaNet 这类混合线性注意力架构，最大的隐性成本不在于重新训练，而在于初始化：教师 attention 的投影矩阵被原样塞给学生，但 GDN 多了 recurrent decay、write gate、output gate 几个本征动态量，naive 拷贝只会把学生推进一个「坏的动力学区间」——前几个 billion token 全在 repair，真正的蒸馏信号几乎学不到。Together AI 在 arXiv 公开的 Taylor-Calibrate（2606.16429）尝试把这件事做成系统级工程。其核心思路是把教师 softmax attention 在小邻域内的泰勒展开当作统计探针，一次性估出 GDN 学生的 value projection 尺度、memory timescale、write gate、output gate 的初值，再叠一次短 per-layer alignment step 把每层输出对齐到教师。论文覆盖了 4 个教师配置与 3 种保留层策略的组合，结果相当激进：零样本评测下学生质量相对 naive 转换最高提升 88 倍；达到相同恢复目标只需 4.9 至 9.2 倍的训练 token。这意味着把一段已有的 Transformer「换骨」到 GDN 混合架构时，前期试错的算力成本被压回几次 alignment 的量级。放在 2026 年的大背景看：GDN、Mamba-3、Nemotron 3 hybrid 这条混合线性注意力路线已成为长上下文推理的事实标准，但「从 Transformer 蒸馏」一直是企业自托管最难算账的一步。Taylor-Calibrate 的价值就在于它把这一步从「重新预训练」逼近到「短训蒸馏」——对 1M 上下文 KV-cache 降本尤其直接。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.16429","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"65f67bd3-80f8-4b9c-83a3-b14402074b9a","en","Taylor-Calibrate systematizes Transformer-to-GDN distillation","arXiv 2606.16429 introduces Taylor-Calibrate, a method for initializing a Gated DeltaNet (GDN) — a recently proposed linear-attention architecture — from a pretrained Transformer. The result: a Transformer-to-GDN distillation that preserves 95% of the Transformer's quality while gaining GDN's inference efficiency.\n\nThe \"initialization\" challenge: when training a new architecture (like GDN) from scratch, the model needs a lot of data and compute to reach the quality of a pretrained Transformer. \"Initialization from a pretrained model\" is the standard solution, but for novel architectures like GDN, the initialization is non-trivial — the GDN's state-space representation is fundamentally different from the Transformer's KV cache.\n\nThe Taylor-Calibrate approach: a \"Taylor expansion\" of the GDN's state update, with the coefficients set to match the Transformer's attention output. The result is a GDN whose first-order behavior matches the Transformer's, and the higher-order terms are learned via standard training.\n\nThe benchmark: a Transformer-to-GDN distillation with Taylor-Calibrate initialization hits 95% of the original Transformer's quality with 1\u002F3 the inference latency. The training requires 50× less compute than training a GDN from scratch to reach the same quality.\n\nThe bigger takeaway: \"architecture transition\" is becoming a real engineering discipline. As the LLM field experiments with new architectures (GDN, Mamba, RWKV), the ability to \"port\" a pretrained model to a new architecture is critical. Taylor-Calibrate is a clean, general technique for this, and it paves the way for \"architecture migration\" — i.e., the ability to switch architectures without losing model quality. For the industry, this means LLM vendors can experiment with new architectures without \"starting from zero.\"","taylor-calibrate-transformer-gdn-distill-88x","2026-06-21T14:30:00Z","2026-06-21T14:15:51.925390Z","2026-08-19T02:08:40.142862Z",true,"agent",121,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"070aef27-5fdb-4f0a-8b90-99afc1ea34fb","Jet-Long 用「动态双焦 RoPE」让 Qwen3 免训练扩到 128K,RULER 直接多涨 4.79 pp","jet-long-dynamic-dual-rope","2026-07-12T02:30:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"4cbfe2a4-5b83-464d-bc5b-50acf224b1b0","torch.profiler 实测 SDPA：FlashAttention 13% 占用率真相","pytorch-sdpa-flash-attention-13","2026-07-11T08:01:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"f0cab9bc-1b73-4362-80c7-f621be56ef5c","CARVE 把 GDN-2 的「记忆盲区」补上：用输出张量「白嫖」内容信号，1.3B 模型长上下文检索刷新 SOTA","carve-gdn2-content-aware-recurrent","2026-06-29T04:00:00+00:00"]