While the industry's compute chases ever-larger decoders, the dmis-lab team did something contrarian: they re-audited the BERT-era encoder pretraining recipe with ablations totaling more than 70 trillion training tokens. The verdict is uncomfortable — the old recipe has been burning GPUs in the wrong places. The result is UltraBERT (150M/380M, Apache 2.0, native 32K context), the first single encoder to top both leaderboards: 91.0 GLUE and 47.7 BEIR for the 380M model.

The Old Encoder Problem: Single-Board Champions

Encoders remain the workhorse of cost-effective text processing: RAG pipelines embed documents with them, and pretraining datasets like FineWeb-Edu were quality-filtered by one. An encoder reads input once, bidirectionally, and returns a vector or score — no token-by-token generation, hence cheap inference. The chronic flaw: the model ranking first on GLUE was never the one ranking first on BEIR, and no single model led both. Architecturally, encoders caught up over the past two years (RoPE, SwiGLU, alternating local/global attention), but the training recipe still followed conventions summarized in the 10B-100B token era.

Three Lessons Bought With 70 Trillion Tokens

The team trained encoders at 150M, 380M and 1.3B on the same data mixture, fine-tuning and evaluating every checkpoint identically. The ablations add up to more than 70T tokens, with the longest single run reaching 8.8T. Three findings:

First, decoders keep scaling; encoders saturate. GLUE and SQuAD plateau around 1-2T tokens, BEIR even earlier, while training and validation loss keep dropping all the way to 8.8T. Loss falls, downstream does not — the extra compute buys almost nothing. Past saturation, changing the recipe beats adding tokens.

Second, the masking ratio is split. Lower ratios (around 5%) benefit classification while higher ones (around 50%) benefit retrieval, contradicting prior work suggesting larger ratios for larger models.

Third, distillation is not universal. Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones.

The Fix: One Encoder, Two LM Heads

Since 5% and 50% each excel at different tasks, the obvious move is training on both. The team tried ratio decay and uniform per-batch sampling — both fail. Uniform sampling just behaves like its average, and decay never recovers low-ratio classification quality (88.8 vs 89.5 GLUE). Looking inside the model revealed why: input embeddings at 6% and 50% masking look nearly identical, but the LM heads need completely different shapes — sharing the output layer forces the two objectives to fight.

So the UltraBERT recipe is LM head decoupling: each batch mixes 5% and 50% masked sequences at 4:1, the encoder body is shared, and each ratio gets its own LM head. One 150M encoder then reaches 89.4 GLUE and 45.1 BEIR — matching the best single-ratio model on classification (89.5) while beating the best one on retrieval (45.0 vs 45.0 for the 50%-only model). Cascade distillation follows: the 1.3B teacher is width-pruned to 380M and distilled, then pruned to 150M and distilled again, with both students learning directly from the 1.3B teacher; the 5% head trains on the teacher's logits with a KL loss, the 50% head with standard MLM. Training runs on 2T tokens, the last 100B at 32K context, and a 2.5% slice of instruction data lets the base follow MCQA prompts zero-shot.

The Scoreboard: Nine of Ten Benchmarks

In the official evaluation, UltraBERT takes the top score on 9 of 10 benchmarks at both sizes, with the biggest jumps on reasoning-style retrieval: +7.3 and +8.3 on RAR-b Code and Math at base size. The 380M model hits 91.0 GLUE and 47.7 BEIR, which the authors describe as, to their knowledge, the first encoder to achieve state of the art on both classification and retrieval. On long-context reliability, the passkey test holds 99% across the full 32K window. The companion UltraBERT-Embed-150M, trained in two contrastive stages (about 500M weakly supervised text and code pairs, then 5.7M curated examples with mined hard negatives), scores 67.24 on MTEB(eng, v2) — the highest of any dense model under 200M parameters.

So What

The real value here is not the leaderboard but the compute-allocation map: encoders top out at 1-2T tokens, and decoder scaling laws do not transfer. Any production line still using encoders for embedding, filtering, or quality control should re-run the budget against the saturation point. A second signal sits in the team's own blog: decision models (the Jev style of "read once, choose from fixed options, generate nothing") are making the encoder's core skill fashionable again — connect that to Microsoft's Qwen3.5-9B-based Decision-1, and the encoder revival is not nostalgia but new cost structure talking. As generative models get more expensive, not generating is itself a competitive edge.

Reference: arXiv:2610.15028 (huggingface.co/blog/dmis-lab/ultrabert)