ALiBi's Hidden Numerical Failure: Longer Contexts Can Make Attention Heads Go Blind

ALiBi is often treated as a cheap, parameter-free positional encoding that supports length extrapolation. A paper submitted to arXiv on August 4 identifies an easily overlooked numerical failure: as context distance grows, ALiBi's linear bias can push softmax exponentials outside the range representable by floating-point formats, turning some attention weights into exact zeros. The authors describe the affected heads as becoming partially blind. Read the paper.

The formula is not the problem; finite precision is

ALiBi adds a head-specific negative bias to attention logits according to the distance between tokens. The farther apart two tokens are, the more negative the bias becomes. In exact mathematics, the resulting weight can approach zero indefinitely. In fp32 or bf16 arithmetic, however, an exponential below the smallest representable positive number underflows. The weight is no longer merely tiny; it becomes exactly zero.

The paper's simplified analysis places the underflow thresholds for fp32 and bf16 at approximately -103.27 and -92.18, respectively. Because default ALiBi assigns different slopes to different attention heads, heads with steeper slopes cross the threshold sooner. For a 16-head layer using bf16, the authors estimate that at a token distance of 2,048, 36.6% of the attention-matrix entries have crossed the underflow threshold.

The failure has two consequences. First, tokens beyond a head's blindness distance receive zero weight and cannot contribute information to the current token. Second, those zeroed terms also disappear from the softmax denominator, redistributing probability mass over the remaining tokens. The mechanism therefore does more than weaken distant information: it silently changes the full attention distribution.

Standard benchmarks may barely notice

The researchers inspect pretrained ALiBi models including BLOOM, Falcon-RW, and MPT, and run controlled pretraining experiments with 148M-parameter decoder models. The result is uncomfortable: the failure can substantially damage token retrieval while producing only small changes on standard decoder benchmarks. The paper reports differences of 1.6 to 3.4 percentage points on regular downstream tasks, while passkey and needle-in-a-haystack retrieval probes reveal much larger effects.

That gap helps explain why the issue may have remained obscure. A model can still look normal on general question answering, commonsense, and language tasks even after part of its long-range attention capacity has vanished. The paper does not conclude that ALiBi is unusable. Default slopes remain a strong baseline on some retrieval tests, and no mitigation wins every task.

Four mitigation families, but no universal fix

The paper evaluates four training-time strategies:

  • Bias clamping: impose a lower bound on the negative bias so it cannot enter the underflow region;
  • Robust slopes: choose head slopes so their blindness distances cover the intended context range;
  • Log-scaled distance: replace linear distance with its logarithm, delaying underflow dramatically;
  • Soft capping: bound the magnitude of attention logits so extreme values are less likely to push exponentials beyond the threshold.

For out-of-context passkey retrieval, the combination of clamping and log-scaled distance raises AUC from the ALiBi baseline of 0.08 to 0.79, nearly a tenfold increase. Yet default ALiBi still performs best on the paper's other needle-in-a-haystack test. The authors therefore avoid declaring a single winner. They recommend clamping when the desired behavior is already a soft sliding window. If a hard sliding window is acceptable, computation outside the window can be skipped explicitly instead of being performed and then discarded through underflow.

The larger lesson: advertised context length is not usable context

The paper's main value is not a verdict that one positional encoding is broken. It pulls long-context evaluation away from architectural promises and back toward numerical behavior. A model accepting a given number of tokens only proves that the input fits. Whether distant information retains nonzero weight, whether gradients still pass through it, and whether retrieval survives are separate questions.

The authors also state an important limitation: most controlled training experiments use 148M-parameter models, so every empirical observation may not transfer directly to larger systems. The analytical underflow mechanism is grounded, but interactions with other architectures, corpora, and training scales still require study.

The next time a model advertises a longer context window, the sharper question is not how many tokens fit. It is this: at the far end of that window, can the model still truly see the information?