It is late in the server room and you are running full-parameter continual pretraining on an open-weight gated model: loss is going down steadily, the logs are clean, no NaN, no error. A paper posted to arXiv on November 3 says that in this seemingly healthy run, one step in five can be wasted — the entire batch of gradients is silently zeroed before the optimizer ever acts, and the log never prints a word about it.

The problem is not in the optimizer or the data. It sits in rows of the embedding table that have been decayed to almost zero.

The audit: 20 of 38 untied gated checkpoints are affected

The authors (led by Washington University in St. Louis, with Nankai University and others) directly read the embedding matrices of 291 open HuggingFace repositories — 280 embedding tables, 101 weight-tied and 190 untied. Each row with a norm below 10⁻⁶ went through a decode-and-retokenize reachability test; after excluding alignment padding, 92 repositories contain raw near-zero rows and 67 repositories (23.0%) are genuinely affected.

Split by architecture, the distribution is clean: 0% of weight-tied models are affected; untied models overall sit at 67/190 (35.3%); untied Transformers at 47/152 (30.9%) — but they do not fall ill; untied gated models (the Mamba, Gated DeltaNet, xLSTM family) reach 52.6% (20/38).

The paper names names: Nemotron-3-Nano-4B, Falcon-H1-1.5B/7B, Mamba-Codestral-7B, Kimi-Linear-48B-Base, Nemotron-H-4B-Base. The offending tokens are ordinary strings: Telugu and Basque subwords, Cyrillic Mongolian words, CRLF code fragments, the chat tag <|im_user|>, and .

The origin: AdamW washes unseen tokens down to near zero

A language model can tie its input and output tables into one matrix, or store them separately. A token that never appears in pretraining text gets exactly zero gradient on its input row at every step, while its output row receives a non-zero softmax gradient every step. Tied architectures keep the norm near 1 through those output gradients; untied architectures have their input rows shrunk step by step by AdamW's decoupled weight decay — the authors calculate that over million-step training (η=4×10⁻⁴, λ=0.1) the norm shrinks to 10⁻¹², and to 10⁻³⁵ at larger learning rates.

They verified this on OLMoE-1B-7B: all 251 near-zero rows in the checkpoint have zero occurrences in a 364M-token sample of its open training corpus (0/251), in line with the cumulative weight-decay curve. Meanwhile 92% of 979 sampled near-zero tokens (901 of 979) keep output norms above 0.3× the vocabulary median — inference is not touched by this pit. That is why nobody noticed: deployment looks fine, training bleeds.

The mechanism: gated readout turns small norms into a 10³⁰ amplifier

Why are Transformers safe? At non-initial positions, Layer-1 attention averages in normal-scale values from earlier tokens and restores the residual stream in one layer; the paper measured input gradients of just 56–98 on Llama-3.1-8B.

Gated models take another path: in Mamba and Gated DeltaNet the historical state readout is multiplied by a SiLU gate, and in xLSTM by a linear query — both current-token factors are exactly zero at zero input. Once the input is near zero, readout follows it down, the residual stream stays below the √ε floor across consecutive layers, and Pre-RMSNorm degenerates into division by √ε: at ε=10⁻⁶ backward gradients are amplified by 10³ per layer, compounding past 10³⁰ across depth.

By sweeping input norms down manually, the authors measured each architecture's amplification ceiling: 1.3×10³⁰ on Falcon-Mamba-7B, 1.0×10³² on xLSTM-7B, and a peak of 4.5×10³⁴ on Nemotron-3-Nano-4B (2.1×10³⁵ on native tokens); the parallel-hybrid Falcon-H1-1.5B has attention as a per-layer backstop, holding spikes near 3.5×10⁶.

Silent zeroing: FP32 overflow, no NaN, no error

The paper's overall measurement: such tokens amplify input-embedding gradients in gated models by over 30 orders of magnitude, up to 35. Those astronomical values are finite and do not blow up training by themselves; the kill happens one step before the optimizer, in gradient clipping. PyTorch's clip_grad_norm_, HuggingFace Trainer, and FSDP2 sum squared gradients in FP32, whose ceiling is about 3.4×10³⁸ — any gradient entry above 1.84×10¹⁹ (√FLT_MAX) overflows to +∞ when squared; in IEEE-754, finite divided by ∞ is exactly 0.0, so the whole batch of gradients becomes zero, with no NaN and no exception. Megatron-Core behaves differently: it detects and aborts.

The chain reaction is worse: on a zeroed step, AdamW advances just stale momentum and weight decay, while the newly triggered near-zero row (momentum and second moment both 0) gets no update, shrinks further under weight decay, and overflows again on the next encounter — a self-locking loop.

On trigger rates, the paper sampled 8,000 documents across 15 corpora and 14 models: English web, SFT, code, and math corpora all show a 0.0% batch trigger rate; multilingual and template workloads are the danger zone — 10.6%–18.8% on Aya, 70.5%–97.4% on Thai, 95.4% on Nemotron-3-Nano-4B Telugu, 100% on Falcon-H1 Mongolian, and 100% at sequence position 0 for Kimi-Linear's <|im_user|> tag. The full funnel in the 1,000-step Telugu run (16,384 tokens per step): 62 rows (0.047% of the vocabulary, 32.3 ppm of tokens) trigger 22.5% of batches, 88.1% of which overflow, leaving 19.8% of all steps with the entire batch gradient zeroed.

The fixes: three lines of GroupClip, or GateFloor for pretraining

For released checkpoints the paper proposes GroupClip: since the amplification is confined to the input embedding while backbone gradients stay at 1–10, clip the two separately and compute the embedding's norm in 64-bit — three lines of code. On the Telugu run, stalled steps drop from 198.3±9.9 to 0 and perplexity goes 13.44→13.04, matching the Oracle Mask that requires a vocabulary pre-scan; all 30.7 encountered near-zero rows self-heal on initial contact. On Mongolian: 11.7→0 and perplexity 15.68→15.58. On H100, step time and peak memory match the baseline.

For pretraining from scratch, the paper proposes GateFloor: add a positive offset to the recurrent gate, SiLU(z)+δ with δ∈[0.01,0.1], so the gate stays positive at zero input. At the 370M scale, the gradient at r=10⁻³⁰ drops from 1.6×10²⁸ to 0.13 — a cut of 28 orders of magnitude; throughput is 542.8k vs 527.2k tokens/s, and validation perplexity stays within ±0.17 of the baseline.

Who should care

The paper's own boundary checks are worth copying: rank-16 LoRA freezes the input embedding and is unaffected — 0 overflows across 9 checkpoints; on-policy RL rollouts decode through the output side and sampled zero near-zero tokens across 16,384 Mongolian and Telugu tokens; but full-parameter RL still triggers whenever a prompt carries a near-zero template tag. The limitations are plain: audits and gradient probes reach 48B, continual pretraining covers 1.5B–4B, and GateFloor from-scratch runs reach just 370M with 1–2 seeds each.

So, the takeaway: if you are fine-tuning or continually pretraining an untied gated model, especially on multilingual or template-heavy data, run a norm pass over the embedding table before you start — rows below 10⁻⁶ are landmines. Stop trusting the idea that no NaN means no problem: the failure that bites in training does not necessarily scream; gradients being zeroed never prints a single log line.