The mainstream recipe for stronger small models is to read more: models with a few billion parameters now routinely train on more than ten trillion tokens. But high-quality English text is finite, so growing budgets end up re-reading the same data with diminishing returns. The Lara team at Translated ran the arithmetic the other way: instead of reading English for the Nth time, spend part of the budget on the text a model has seen least — text in other languages.
The backdrop: non-English data below 12%
The paper first audits the field: across the open models it reviews, non-English pretraining data tops out at a 12% share. Models built for European languages give far more — Teuken-7B at 51%, Apertus at 40%, EuroLLM at 37% — yet, the paper notes, they score well below general-purpose models of the same size. A long-standing worry adds friction: in small models, languages compete for capacity.
The Lara team's answer changes the data mixture and the architecture together.
The model: 7.2B total, 0.6B active
Lara Mini base is pretrained from scratch on 6T tokens: English is only 41%, while 44 other languages — covering all 24 official EU languages, with Simplified and Traditional Chinese counted separately per the model card — total 2.18T, a 36.4% share. The balance shifts as training proceeds: English falls from 44% of the main stage to 27% of the anneal, while the other languages grow from 46% to 55% and outweigh English in the anneal. Of the 6T tokens, 97% are a first read of a document, so repetition barely props up the budget.
The architecture serves both multilinguality and inference cost. Every layer is a fine-grained MoE that activates 4 of 128 routed experts per token, plus one shared expert; total parameters are 7.2B, active just 0.6B. Only 7 of 34 layers keep global attention — the rest interleave Gated DeltaNet and sliding-window attention with a window of 512, so the inference state does not grow with input length. A fourth stage of 100B tokens then extends context from 4,096 to 128k. Note this is a base model: it continues text, with no instruction tuning and no safety tuning.
The scorecard: 57.0, self-reported
Among the 36 open base models with up to 2B active parameters that the paper evaluates itself, Lara Mini base averages 57.0 across English, multilingual, math and code, 3.1 points above granite-4.0-h-tiny; its multilingual average is 56.7, 3.4 points above Gemma-4-E2B; it is first on math, second on English, fifth on code. The cost contrast is stark: granite-4.0-h-tiny and Qwen3-1.7B read 22.5T and 36T tokens, about 4 and 6 times as many. Across all 56 compared models, only six score higher overall, all with 4 to 14 times its active parameters.
Inference is the other half of the pitch: on a single A100, single user, no quantization, it generates about 1,100 characters per second — on par with dense models of about 1B parameters — while every higher-scoring model generates at least 1.8 times more slowly. The closest, MiniCPM5-2B, is 0.5 points higher overall but 8.7 points lower on the multilingual average.
How languages split the experts
The mechanism finding may be the most interesting part: measuring, for each language, the Jensen–Shannon divergence between its distribution over the 128 routed experts and the mean over supported languages, the paper finds that languages route differently in the first and last layers and almost identically in the middle layers. MoE lets languages take different doors; the authors argue this reduces capacity competition between languages — a direct answer to the old worry.
So what
Weights are on Hugging Face under CC BY-NC-SA 4.0: free for research and non-commercial use, commercial use to be negotiated separately. The team also promises T-HellaSwag, a machine translation of the HellaSwag commonsense benchmark into 110 language variants (105 languages, 25 scripts, 200 documents per variant and 22,000 in total), claiming higher quality than the commonly used multilingual version; at paper submission it was still marked as forthcoming.
For anyone building Chinese-language models, the value here is methodological: the data mixture was tuned as a first-class design variable. Language diversity is not a tax to pay but a mine not yet exhausted. As marginal returns on English corpora thin out, where the next token comes from may matter more than how many more parameters you stack.
Paper: https://arxiv.org/abs/2611.16956
Weights: https://huggingface.co/translated/Lara-4-Mini-Base