While every major open-weight lab converges on decoder-only transformers, Yandex just shipped the opposite bet: it open-sourced AliceAI-T5-35B-A0.6B, the production backbone of Alice AI answers in Search, to Hugging Face. This is a from-scratch encoder-decoder (T5-style) model with 34.35B total parameters and only ~0.6B active per token — the last serious encoder-decoder MoE efforts from major teams (NLLB-MoE, Switch Transformer, FLAN-MoE) date back to 2021-2023. (HF model card)

Why revive an "outdated" architecture

The answer sits in the product shape. According to Yandex's engineering blog, this model powers the "quick answers" card on the Search results page: upstream models retrieve documents, an 80M-parameter extractor compresses them into "infocontexts", and this model writes the final answer — all within seconds at peak load. That read-long-write-short RAG profile is exactly where encoder-decoder traditionally shines: each MoE layer holds 512 experts with top-8 routing per token, the encoder processes retrieved documents in parallel, cheaper than causal attention, while sparse MoE keeps capacity at the 35B level with per-token compute near a sub-billion model. It was pretrained with a UL2 denoising objective on 15T tokens, with context extended from 8K to 128K via YaRN late in training. (Habr engineering blog)

Benchmarks: a specialist, not an all-rounder

The official table pits it against Qwen3.5 2B/4B/35B-A3B, Gemma 4 E4B, and T5 Gemma 2 base models. The pattern is crisp. It sweeps factuality and extraction: 68.0 on the Russian CultCat benchmark, 81.3 on WikiWebFacts, and 94.7 on Ruler 32K — all beating Qwen3.5-35B-A3B despite that model activating 5x more parameters per token. But math and code (MATH 500 at 62.9, HumanEval at 69.3) and Ruler 128K (81.4) still favor Qwen3.5-35B-A3B. Blind side-by-side evaluations tell the same story: 77% win rate over fine-tuned Qwen3.5-2B, 54% over 4B, 69% over T5 Gemma 2.4B-4B — but only 44% against Qwen3.5-35B-A3B, which the team itself notes is "significantly more expensive at inference".

Two training signals worth noting

First, routing stability: the team started with a Macro-LBL-style auxiliary-loss balance, saw the encoder-side routing collapse at the 15T-token scale, and switched to DeepSeek-V3's aux-free routing (bias terms updated without gradients) to stabilize — another datapoint that auxiliary-loss balancing breaks at scale. Second, the optimizer: they chose Muon over AdamW and Adan, reporting better results in their experiments. One caveat easy to miss: external users get Hugging Face Transformers inference only; the optimized production inference stack stays inside Yandex. The company reports +0.34% sessions-per-user in online A/B testing and a 56.7% SbS win rate against Google AI Overview.

For teams building retrieval-augmented or structured-extraction workloads, this is a rare non-decoder-only reference point. For general-benchmark watchers, it was never meant for you — a specialist architecture winning its niche is itself the lesson: model design still follows product shape, even in the MoE era.