Dropout was standard regularization in the 2017 Transformer paper, yet it has all but vanished from large language model pretraining. GPT-3 and OPT kept it; PaLM used it only during finetuning; LLaMA-style models no longer document it at all. The reasoning sounds solid: single-epoch pretraining over trillions of tokens leaves little room for classical overfitting, and some studies report that dropout degrades accuracy in this regime. A new paper from the Cerebras Systems team, "Don't Drop Dropout" (an extended version of their ICML 2026 paper, arXiv:2609.05275), argues the opposite: what got abandoned may not be dropout itself, but misconfigured dropout.
2400+ Experiments Reframe Layer Dropout
The study focuses on layer dropout, also known as stochastic depth: during training, entire transformer blocks are skipped with some probability. Unlike neuron-level sparsity, skipping whole layers is structured sparsity, and the FLOPs saved scale almost linearly with the dropout rate. The team ran more than 2400 pretraining experiments on Cerebras CS-3 systems, spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, at a compute-optimal budget of 20 tokens per parameter, using the Celerity-style architecture (ALiBi position embeddings, squared ReLU activations, Llama3 vocabulary).
The core result is a combined recipe: increase the dropout rate across depth (drop less in shallow layers, more in deep layers), decrease it over training time (high initial noise for exploration, then convergence), and re-tune optimizer hyperparameters for each dropout rate. With this configuration, layer dropout reaches lower loss at the same training FLOPs; for a fixed number of steps, it saves up to 25% of training FLOPs while matching or beating the validation loss. The paper reports that at 5% FLOPs savings on 503M and 906M models, the dropout configuration beat the dense baseline, which the authors describe as the first demonstration that fewer training FLOPs can outperform a full-compute baseline. On sampling granularity, per-sequence dropout yields lower loss than per-batch dropout, a comparison the authors say they are the first to systematically study.
"Free" Elasticity at Inference Time
Models pretrained with layer dropout gain depth elasticity for free: zero-shot early exit or intermediate layer skipping degrades loss gracefully, while dense models spike almost immediately. One large model can thus shrink toward smaller-model performance levels to fit latency budgets, without retraining. Combined with post-training tricks like self-speculative decoding, inference speeds up by up to 1.5x with negligible accuracy loss. The layer distribution carries a clear trade-off: the alternating distribution (ALD, dropping every other layer) is most robust to layer skipping, while the increasing distribution (ILD) gives better base accuracy and early-exit behavior; the paper recommends ILD as models scale.
So What
This work reframes "dropout is useless at scale" as a configuration problem. Layer dropout is no longer just an anti-overfitting regularizer; it is a structured-sparsity switch that saves training compute and buys inference elasticity. For teams with tight pretraining budgets, 25% of FLOPs is a serious line item. But keep the boundaries in mind: experiments top out at 8.2B parameters, so frontier-scale behavior is unverified; conclusions rest on the Celerity-style decoder-only architecture and a 20 tokens-per-parameter budget; and the numbers are self-reported on the team's own hardware, awaiting independent replication. The next time a lab's pretraining bill drops unexpectedly, check whether dropout quietly returned to the recipe.