Inference stacks quietly depend on a small prediction: before the model emits its first token, a scheduler wants to know how long the reply will be, because continuous batching uses that estimate to group requests. The dominant recipe trains a regressor on prefill-phase hidden states and outputs a length before decoding starts. The method list has grown; the numbers have not converged. The paper "On the Fundamental Limits of Prefill-Based Decoding Length Prediction" (arXiv:2610.40541, posted October 29) opens with an uncomfortable observation: the same predictor architecture reports MAE around 80 on ForeLen's long-sequence split with Qwen2.5-7B, degrades past 150 on ShareGPT with Qwen3-32B, and past 260 on RL-sampled data — a three-fold swing with the predictor untouched.
The ceiling is drawn by a normal distribution
The authors locate the root cause inside autoregressive sampling itself: decode the same prompt twice and the lengths differ. They model decoding as a stopped stochastic process. Each step carries a stopping probability that is not constant but ramps up over hundreds of steps; because response length accumulates from many segment-level local decisions, a central limit theorem kicks in, making the per-prompt decode length asymptotically normal with an explicit error term. The variance obeys an affine law σ=αμ+b, whose slope α approximately equals the coefficient of variation of the number of response segments. A counterfactual sharpens the point: if the stopping probability were constant, length would be geometric, σ/μ would tend to 1, and every predictor would face an irreducible relative MAE of about 74%; the measured σ/μ≈0.15 rules that darker world out. The validation is thorough: 300 ShareGPT prompts, each decoded 200 independent times with Qwen3-32B (temperature 0.6, top-k 20, top-p 0.95); two of the 300 are fully deterministic (all 200 decodes return the same 8-token answer). Across the remaining 298, the Gaussian value σ√(2/π) reproduces the per-prompt MAE to within 1.9%, the central mass to within two percentage points, and the residual departures (median skewness +0.27, excess kurtosis +0.35) point the way the sampling-noise correction predicts.
Which models drift
Table 1 fits (α, b) per model on ShareGPT at 300 prompts × 200 decodes. Qwen3-4B-Instruct is the most stable (α=0.1068); Qwen3-4B with thinking disabled drops to 0.2029; Llama3-8B sits at 0.2042; Qwen3-32B at 0.1486. The most telling row is DeepSeek-R1-distilled-thinking: a modest slope (0.1156) paired with an unusually large intercept (147.6), which the authors read as the exploratory randomness intrinsic to chain-of-thought generation — even when the final answer is short, the reasoning path length swings widely. Different variance means different ceilings: Llama3-8B averages σ of 112, while Qwen3-30B-A3B reaches 261.
After a fair retest
With the model (Qwen3-32B), the data (roughly 18,000 ShareGPT prompts for training, IDs 18000–18300 as the 300-prompt test set), and the sampling parameters all pinned, eight predictors share one stage. The simplest, MLP-Concat (concatenating the last-token hidden vector of all layers, roughly 100k dimensions, into a 3-layer MLP), scores MAE 169.31; adding LayerNorm improves it to 168.13, the best of the eight. Three MoE variants land between 170.70 and 177.91; Soft-Label and FiLM hover around 180. EGTP-style entropy-guided pooling — the method that reported up to 29.16% MAE reduction over baselines on ForeLen — collapses to 466.79 here, far behind even the plain MLP-Mean at 181.21. The authors' explanation: when the token-level entropy distribution does not align with the length signal, entropy-weighted aggregation injects noise. The theoretical floor is settled in the same table: the 300 test prompts average σ of 162.3 (median 133.9; 10% of prompts exceed 300), which puts the Gaussian noise floor at 129.5; the best predictor at 168.1 exceeds it by 38.6 tokens, and the paper proves that surplus is at most 30% of the system error. For tolerance metrics, the Acc@100 ceiling is 57.9% while the measured best is 48.9% — 8.9 points short; reaching a 90% Acc@100 would require pushing σ below 61. Cross-domain results hug the same line: training on LMSYS-200K and testing on ShareGPT yields MAE 182.40; training on LeetCode and testing on HumanEval yields 29.28 with Acc@100 at 96.28% — align the domain and the task gets easy, misalign it and you sit on the floor.
So what: read σ before buying a predictor
The conclusion is blunt: prefill-side length prediction is a noise-limited problem where system error dominates model error, and architectural elaboration buys marginal or negative returns. Improvement has two honest paths — control the variance of generation itself (lower temperature, constrained decoding, prompt design), or re-estimate iteratively once decoding has begun (updates every 50–100 tokens; in the every-token limit the remaining length collapses to a deterministic countdown and the MAE bound tends to zero, at the price of an encoder forward pass per update). For benchmarks, the authors recommend fixing model, data, and sampling, reporting per-prompt σ, and treating relative error as the currency. The paper also discloses its own weaknesses: features and labels came from different workers of a PD-disaggregated pipeline with batching and numerical behavior not strictly controlled, and training labels are single decodes that carry noise of their own. Code is open-sourced under MIT on GitHub; the paper is at arXiv:2610.40541. Next time a decode-length predictor posts a new low on a leaderboard, ask first: what is σ in that setup?