[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-near-zero-embeddings-gated-training-stall":3,"topics-all":38,"news-related-ba729d8d-d18e-4e77-824e-1db99ec469eb":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"ba729d8d-d18e-4e77-824e-1db99ec469eb","近零词嵌入暗坑:五成门控检查点训练静默白跑","华盛顿大学与南开团队普查291个开源仓库:52.6%的未绑定门控检查点藏着范数近零的词嵌入,病根是AdamW权重衰减把没见过的token洗到近零。这类token在门控模型里把输入梯度放大三十多个数量级,撑爆FP32裁剪,泰卢固语续训19.8%的步数整批白跑;推理照常,坑只在训练侧,三行GroupClip就能修复。","深夜的机房里,你正在给一个门控架构的开源模型做全参数续训:loss 稳步下降,日志干净,没有 NaN,没有报错。[11 月 3 日挂上 arXiv 的一篇论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04449)说,这种看起来一切正常的训练,可能每五步就白跑一步——整批梯度在优化器动手之前就被静默清零,而日志里一个字都不会提。\n\n问题不在优化器,也不在数据,而在嵌入表里一些被洗到接近零的行。\n\n## 普查:38 个未绑定门控检查点,20 个带病\n\n论文作者(华盛顿大学圣路易斯分校牵头,南开大学等参与)直接读取了 291 个 HuggingFace 开源仓库的嵌入矩阵,共 280 张嵌入表,101 个权重绑定、190 个未绑定。范数低于 10⁻⁶ 的每一行都做解码再重分词的可达性测试,排除对齐填充后:92 个仓库有原始近零行,67 个仓库(23.0%)真正带病。\n\n按架构拆开,分布很干净:权重绑定模型 0% 中招;未绑定模型整体 67\u002F190(35.3%);未绑定 Transformer 占 47\u002F152(30.9%),但它们不发病;未绑定门控模型(Mamba、Gated DeltaNet、xLSTM 这一族)达到 52.6%(20\u002F38)。\n\n论文点名的中招例子:Nemotron-3-Nano-4B、Falcon-H1-1.5B\u002F7B、Mamba-Codestral-7B、Kimi-Linear-48B-Base、Nemotron-H-4B-Base。出事的 token 全是普通字符串:泰卢固语、巴斯克语子词,西里尔蒙文词,CRLF 代码碎片,聊天标签 \u003C|im_user|>,还有 \u003Cs>。\n\n## 病根:AdamW 把预训练没见过的 token 洗到近零\n\n语言模型的输入表和输出表可以共用一张矩阵(绑定),也可以分开存(未绑定)。一个 token 若在预训练语料里从未出现,它的输入行每步梯度恒为零,输出行却每步都能从 softmax 拿到非零梯度。绑定架构靠输出梯度把范数撑在 1 附近;未绑定架构的输入行则被 AdamW 的解耦权重衰减逐步收缩——作者测算,百万步训练(η=4×10⁻⁴、λ=0.1)后范数缩到 10⁻¹²,学习率更大时到 10⁻³⁵。\n\n他们拿 OLMoE-1B-7B 做了实证:检查点里的 251 个近零行,在 364M token 的开源训练语料抽样中全部零出现(0\u002F251),与权重衰减的累计曲线对得上。而 979 个近零 token 里 92%(901 of 979)的输出范数仍在词表中位数的 0.3× 以上——推理不受这个坑影响。这就是它长期没被发现的原因:部署侧安好,训练侧失血。\n\n## 机制:门控读出把小范数变成 10³⁰ 放大器\n\nTransformer 为什么不怕?在非首位置,第 1 层注意力能把前文正常尺度的 value 平均进来,一层就把残差流拉回正常;论文实测 Llama-3.1-8B 的输入梯度只有 56–98。\n\n门控模型走的是另一条路:Mamba、Gated DeltaNet 的历史状态读出要乘一个 SiLU 门,xLSTM 要乘一个线性 query,这两类当前 token 因子在零输入处都恰好为零。输入近零,读出跟着趋零,残差流连续多层低于 √ε 地板,Pre-RMSNorm 退化成除以 √ε:ε=10⁻⁶ 时反向梯度每层放大 10³,跨层叠加冲破 10³⁰。\n\n作者用人工下扫输入范数的方式量出各架构的放大上限:Falcon-Mamba-7B 为 1.3×10³⁰,xLSTM-7B 为 1.0×10³²,Nemotron-3-Nano-4B 峰值 4.5×10³⁴(原生 token 上 2.1×10³⁵);并行混合的 Falcon-H1-1.5B 每层都有注意力兜底,尖峰压在 3.5×10⁶。\n\n## 静默清零:FP32 溢出,无 NaN 也无报错\n\n论文的总体测量是:这类 token 让门控模型的输入嵌入梯度放大超过 30 个数量级,最高到 35。这些天文数字本身是有限值,不会直接炸掉训练;动手的是优化器前一步的梯度裁剪。PyTorch 的 clip_grad_norm_、HuggingFace Trainer、FSDP2 都在 FP32 里累加梯度平方,FP32 上限约 3.4×10³⁸——任何梯度分量超过 1.84×10¹⁹(即 √FLT_MAX),平方就溢出成 +∞;IEEE-754 里有限值除以 ∞ 恰等于 0.0,于是整批梯度被清零,没有 NaN,也没有异常。Megatron-Core 的行为不同:检测后直接报错中止。\n\n更麻烦的是连锁反应:清零的那一步,AdamW 只推进陈旧动量和权重衰减;新触发的近零行(动量、二阶矩都是 0)拿不到更新,还被权重衰减继续缩小,下次遇到再次溢出,形成自锁。\n\n触发率方面,论文在 15 个语料、14 个模型上抽样 8,000 篇文档:英文网页、SFT、代码、数学语料的批次触发率都是 0.0%;多语言和模板数据是重灾区——Aya 为 10.6%–18.8%,泰语为 70.5%–97.4%,Nemotron-3-Nano-4B 泰卢固语 95.4%,Falcon-H1 蒙古语 100%,Kimi-Linear 的 \u003C|im_user|> 标签在序列首位同样是 100%。1,000 步泰卢固语续训(每步 16,384 token)的完整漏斗:62 行(占词表 0.047%、占 token 32.3 ppm)触发 22.5% 的批次,其中 88.1% 溢出,最终 19.8% 的步数整批梯度归零。\n\n## 修法:三行 GroupClip,或预训练加 GateFloor\n\n对已有检查点,论文给出 GroupClip:既然放大只发生在输入嵌入里,主干梯度保持在 1–10,那就把两者分开裁剪,嵌入的范数改用 64 位计算——三行代码。泰卢固语实验:停滞步数从 198.3±9.9 降到 0,困惑度 13.44→13.04,追平需要预先扫词表的 Oracle Mask;遇到的 30.7 个近零行初次接触即自愈。蒙古语实验:11.7→0,困惑度 15.68→15.58。H100 上步时与峰值内存和基线持平。\n\n对从头预训练,论文给出 GateFloor:给循环门加一个正偏置,变成 SiLU(z)+δ,δ∈[0.01,0.1],让门在零输入处仍大于零。370M 参数规模下,r=10⁻³⁰ 的梯度从 1.6×10²⁸ 降到 0.13,砍掉 28 个数量级;吞吐为 542.8k 对 527.2k token\u002Fs,验证困惑度与基线差距在 ±0.17 内。\n\n## 谁该在意\n\n论文自测的边界值得抄:rank-16 LoRA 冻结输入嵌入,不受影响,9 个检查点 0 次溢出;on-policy RL 的 rollout 走输出侧,16,384 个蒙古语与泰卢固语 token 采样零命中;但全参数 RL 只要 prompt 带近零模板标签照样中招。局限也很直白:审计和梯度探针覆盖到 48B,续训只测了 1.5B–4B,GateFloor 从头训只到 370M,每组 1–2 个种子。\n\n所以呢:如果你正在对未绑定门控模型做全参数微调或续训,喂的又是多语言或模板化数据,开训前先对嵌入表跑一遍范数统计,低于 10⁻⁶ 的行就是雷。别再信没有 NaN 就没问题这种判断——训练里会咬人的故障,未必会喊疼;梯度被清零这件事,日志一行都不会给你。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04449","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"670b4203-ba62-4e0e-aa2f-3afb0a0df711","en","Near-Zero Embeddings Silently Stall Gated Model Training","Audit of 291 repos: 52.6% of untied gated checkpoints carry near-zero embeddings; FP32 clipping overflows, zeroing 19.8% of Telugu CPT steps.","It is late in the server room and you are running full-parameter continual pretraining on an open-weight gated model: loss is going down steadily, the logs are clean, no NaN, no error. A [paper posted to arXiv on November 3](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04449) says that in this seemingly healthy run, one step in five can be wasted — the entire batch of gradients is silently zeroed before the optimizer ever acts, and the log never prints a word about it.\n\nThe problem is not in the optimizer or the data. It sits in rows of the embedding table that have been decayed to almost zero.\n\n## The audit: 20 of 38 untied gated checkpoints are affected\n\nThe authors (led by Washington University in St. Louis, with Nankai University and others) directly read the embedding matrices of 291 open HuggingFace repositories — 280 embedding tables, 101 weight-tied and 190 untied. Each row with a norm below 10⁻⁶ went through a decode-and-retokenize reachability test; after excluding alignment padding, 92 repositories contain raw near-zero rows and 67 repositories (23.0%) are genuinely affected.\n\nSplit by architecture, the distribution is clean: 0% of weight-tied models are affected; untied models overall sit at 67\u002F190 (35.3%); untied Transformers at 47\u002F152 (30.9%) — but they do not fall ill; untied gated models (the Mamba, Gated DeltaNet, xLSTM family) reach 52.6% (20\u002F38).\n\nThe paper names names: Nemotron-3-Nano-4B, Falcon-H1-1.5B\u002F7B, Mamba-Codestral-7B, Kimi-Linear-48B-Base, Nemotron-H-4B-Base. The offending tokens are ordinary strings: Telugu and Basque subwords, Cyrillic Mongolian words, CRLF code fragments, the chat tag \u003C|im_user|>, and \u003Cs>.\n\n## The origin: AdamW washes unseen tokens down to near zero\n\nA language model can tie its input and output tables into one matrix, or store them separately. A token that never appears in pretraining text gets exactly zero gradient on its input row at every step, while its output row receives a non-zero softmax gradient every step. Tied architectures keep the norm near 1 through those output gradients; untied architectures have their input rows shrunk step by step by AdamW's decoupled weight decay — the authors calculate that over million-step training (η=4×10⁻⁴, λ=0.1) the norm shrinks to 10⁻¹², and to 10⁻³⁵ at larger learning rates.\n\nThey verified this on OLMoE-1B-7B: all 251 near-zero rows in the checkpoint have zero occurrences in a 364M-token sample of its open training corpus (0\u002F251), in line with the cumulative weight-decay curve. Meanwhile 92% of 979 sampled near-zero tokens (901 of 979) keep output norms above 0.3× the vocabulary median — inference is not touched by this pit. That is why nobody noticed: deployment looks fine, training bleeds.\n\n## The mechanism: gated readout turns small norms into a 10³⁰ amplifier\n\nWhy are Transformers safe? At non-initial positions, Layer-1 attention averages in normal-scale values from earlier tokens and restores the residual stream in one layer; the paper measured input gradients of just 56–98 on Llama-3.1-8B.\n\nGated models take another path: in Mamba and Gated DeltaNet the historical state readout is multiplied by a SiLU gate, and in xLSTM by a linear query — both current-token factors are exactly zero at zero input. Once the input is near zero, readout follows it down, the residual stream stays below the √ε floor across consecutive layers, and Pre-RMSNorm degenerates into division by √ε: at ε=10⁻⁶ backward gradients are amplified by 10³ per layer, compounding past 10³⁰ across depth.\n\nBy sweeping input norms down manually, the authors measured each architecture's amplification ceiling: 1.3×10³⁰ on Falcon-Mamba-7B, 1.0×10³² on xLSTM-7B, and a peak of 4.5×10³⁴ on Nemotron-3-Nano-4B (2.1×10³⁵ on native tokens); the parallel-hybrid Falcon-H1-1.5B has attention as a per-layer backstop, holding spikes near 3.5×10⁶.\n\n## Silent zeroing: FP32 overflow, no NaN, no error\n\nThe paper's overall measurement: such tokens amplify input-embedding gradients in gated models by over 30 orders of magnitude, up to 35. Those astronomical values are finite and do not blow up training by themselves; the kill happens one step before the optimizer, in gradient clipping. PyTorch's clip_grad_norm_, HuggingFace Trainer, and FSDP2 sum squared gradients in FP32, whose ceiling is about 3.4×10³⁸ — any gradient entry above 1.84×10¹⁹ (√FLT_MAX) overflows to +∞ when squared; in IEEE-754, finite divided by ∞ is exactly 0.0, so the whole batch of gradients becomes zero, with no NaN and no exception. Megatron-Core behaves differently: it detects and aborts.\n\nThe chain reaction is worse: on a zeroed step, AdamW advances just stale momentum and weight decay, while the newly triggered near-zero row (momentum and second moment both 0) gets no update, shrinks further under weight decay, and overflows again on the next encounter — a self-locking loop.\n\nOn trigger rates, the paper sampled 8,000 documents across 15 corpora and 14 models: English web, SFT, code, and math corpora all show a 0.0% batch trigger rate; multilingual and template workloads are the danger zone — 10.6%–18.8% on Aya, 70.5%–97.4% on Thai, 95.4% on Nemotron-3-Nano-4B Telugu, 100% on Falcon-H1 Mongolian, and 100% at sequence position 0 for Kimi-Linear's \u003C|im_user|> tag. The full funnel in the 1,000-step Telugu run (16,384 tokens per step): 62 rows (0.047% of the vocabulary, 32.3 ppm of tokens) trigger 22.5% of batches, 88.1% of which overflow, leaving 19.8% of all steps with the entire batch gradient zeroed.\n\n## The fixes: three lines of GroupClip, or GateFloor for pretraining\n\nFor released checkpoints the paper proposes GroupClip: since the amplification is confined to the input embedding while backbone gradients stay at 1–10, clip the two separately and compute the embedding's norm in 64-bit — three lines of code. On the Telugu run, stalled steps drop from 198.3±9.9 to 0 and perplexity goes 13.44→13.04, matching the Oracle Mask that requires a vocabulary pre-scan; all 30.7 encountered near-zero rows self-heal on initial contact. On Mongolian: 11.7→0 and perplexity 15.68→15.58. On H100, step time and peak memory match the baseline.\n\nFor pretraining from scratch, the paper proposes GateFloor: add a positive offset to the recurrent gate, SiLU(z)+δ with δ∈[0.01,0.1], so the gate stays positive at zero input. At the 370M scale, the gradient at r=10⁻³⁰ drops from 1.6×10²⁸ to 0.13 — a cut of 28 orders of magnitude; throughput is 542.8k vs 527.2k tokens\u002Fs, and validation perplexity stays within ±0.17 of the baseline.\n\n## Who should care\n\nThe paper's own boundary checks are worth copying: rank-16 LoRA freezes the input embedding and is unaffected — 0 overflows across 9 checkpoints; on-policy RL rollouts decode through the output side and sampled zero near-zero tokens across 16,384 Mongolian and Telugu tokens; but full-parameter RL still triggers whenever a prompt carries a near-zero template tag. The limitations are plain: audits and gradient probes reach 48B, continual pretraining covers 1.5B–4B, and GateFloor from-scratch runs reach just 370M with 1–2 seeds each.\n\nSo, the takeaway: if you are fine-tuning or continually pretraining an untied gated model, especially on multilingual or template-heavy data, run a norm pass over the embedding table before you start — rows below 10⁻⁶ are landmines. Stop trusting the idea that no NaN means no problem: the failure that bites in training does not necessarily scream; gradients being zeroed never prints a single log line.\n","near-zero-embeddings-gated-training-stall","2026-11-04T15:20:36Z","2026-11-04T15:20:46.122346Z","2026-11-04T15:20:46.122357Z",true,"agent",38,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"1fe72544-eeb2-47ad-a849-450ef7056065","Token-Superposition：10B MoE 预训练时长砍掉 2.5 倍","nous-tst-token-superposition-10b-2-5x","2026-05-13T10:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"eb251970-12c7-4edb-8de5-ef33e60f09fa","BYOD：3.2 GPU 小时把自回归模型改成扩散模型","byod-ar-to-diffusion-conversion-lora-single-gpu","2026-11-02T09:30:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"eb55e1cf-15c1-487f-845c-e81f87f11cfa","红白机 CPU 编进 Qwen 权重:没训练,纯编译","nesformer-qwen-nes-cpu-weights","2026-10-31T17:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"41889a7a-a82d-4e8e-b693-7db85a4f62c7","CMU统一训练:数学分打平,下一轮代码反超分阶段","unified-training-single-likelihood-foundation-models","2026-10-31T09:16:57+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"55663822-d49d-4c7e-a7c7-ba9770657c25","清华DBS动态分频:音源分离SI-SDR提升29%","dynamic-band-splitting-audio-separation","2026-10-21T15:02:40+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"a491d4db-2d5a-4723-b093-28d1512fd647","VLA-Δ:物理状态变化当监督信号,LIBERO 98.4","vla-delta-physical-state-transition","2026-10-20T19:07:27+00:00"]