How much does it cost to pretrain a 2B-parameter language model from scratch? Tsinghua University's PACMAN group just published an answer that made the academic community sit up: under 6,900 USD, on consumer-grade RTX 5090 GPUs. The project, called Puro-2B, also open-sources the complete training recipe — data, code, and model weights — under Apache 2.0.

The Cost Wall of Pretraining

Language model pretraining has become almost synonymous with prohibitive cost, putting it out of reach for much of academia and the open-source community. The paper cites two reference points: even at small scale, training Llama-3.2-3B costs over 1.5 million USD, and reproducing SmolLM3-3B needs over 700K. Strong open-source efforts already exist — open-weight models and open training recipes — but a recipe that is cost-efficient, hardware-accessible, and fully open-source has long been missing. Puro-2B fills exactly that gap.

What They Did

Using this recipe, the team trained a collection of Puro-2B models from scratch on consumer-grade RTX 5090 GPUs, feeding up to 1.4 trillion tokens with FP8 precision throughout. Their best model was trained at a compute cost of less than 6.9K USD and approaches Qwen2.5-1.5B performance under their evaluation protocol. The cost efficiency comes from a combination of approaches: hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and a carefully designed data recipe.

The engineering details deserve a closer look too. The training code is built on NVIDIA Megatron-LM (core_v0.16.0), and the public patch preserves a set of battle-tested capabilities: packed NPY pretraining data with resume-safe phase transitions, the MuonHyperball optimizer (with correct AdamW routing for normalization, embedding, bias, output, and other non-matrix parameters), memory-balanced layer-wise distributed optimizer state, blockwise-FP8 with non-persistent checkpoint compatibility, and data-corruption-safe rerun skipping.

Two Extra Findings

Beyond the recipe itself, the paper offers two additional results. First, across the Puro-2B collection, the team derived a Puro Cost Scaling Law relating training cost to average model performance; the fitted law suggests that about 4.4K — under the 5,090 mark — is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, they examined how pretraining data curricula shape downstream performance after post-training. Such controlled studies are only possible with access to the full pretraining pipeline rather than model weights alone — which is precisely the point of going fully open.

So What

The significance of Puro-2B is not how strong a 2B model can be, but that it pushes the entry ticket for from-scratch pretraining down to a scale any lab can afford. Paper: arXiv:2608.27370; full recipe at the HF collection. When reproduction cost drops from millions to a few thousand dollars, pretraining research in academic labs can truly come alive — which link in the chain do you think gets demystified next?