[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-puro-2b-open-pretraining-recipe":3,"topics-all":35,"news-related-936c14f5-53ec-4c86-8a5c-bb2def8b7dff":54},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"936c14f5-53ec-4c86-8a5c-bb2def8b7dff","清华团队 5090 美元从零预训练 2B 模型,完整配方开源","清华 PACMAN 组在消费级 RTX 5090 上以不到 6900 美元从零预训练出 2B 参数模型 Puro-2B,最多喂了 1.4 万亿 token,性能接近 Qwen2.5-1.5B;拟合出的成本缩放律显示约 4400 美元就能追平 Qwen2-1.5B。数据、代码、权重以 Apache 2.0 全开源。","从零预训练一个 2B 参数的语言模型要花多少钱?清华 PACMAN 组用一篇新论文给出了一个让整个学术圈都坐不住的答案:不到 6900 美元,而且是在消费级 RTX 5090 显卡上完成的。这个名为 Puro-2B 的项目还把完整训练配方——数据、代码、模型权重——以 Apache 2.0 协议全部开源。\n\n## 背景:预训练的成本墙\n\n语言模型预训练几乎成了「prohibitive cost」的同义词,对学术界和开源社区来说尤其如此。论文里给了两个参照:即使在小规模上,训练 Llama-3.2-3B 的成本超过 150 万美元,复现 SmolLM3-3B 也要超过 70 万美元。开源社区虽然已有开放权重的模型和开源训练配方,但「成本可控、硬件可及、完全开源」三者兼得的预训练配方一直缺位。Puro-2B 补的就是这个缺口。\n\n## 他们做了什么\n\n团队用这套配方在消费级 RTX 5090 GPU 上从零训练了一组 Puro-2B 模型,最多喂到 1.4 万亿 token,全程使用 FP8 精度。其中最好的模型计算成本不到 6900 美元,在他们设定的评测协议下接近 Qwen2.5-1.5B 的表现。这个成本效率是靠一整套组合拳实现的:硬件选型、低精度训练、hyperball 优化、课程化模型平均(curriculum model averaging),以及数据配方的精细设计。\n\n工程细节同样值得细看。训练代码基于 NVIDIA Megatron-LM(core_v0.16.0)改造,公开发布的补丁保留了一系列实战能力:packed NPY 预训练数据与可断点续训的阶段切换、MuonHyperball 优化器(对归一化层、embedding、bias、输出层等非矩阵参数走正确的 AdamW 路由)、逐层分布式优化器状态的内存均衡、分块 FP8 与非持久化 checkpoint 兼容,以及数据损坏安全的重跑跳过机制。\n\n## 两个额外发现\n\n论文还给出了两条超出配方本身的结论。第一,团队在 Puro-2B 模型集合上拟合出一条「成本缩放律」:训练成本与平均性能直接挂钩,拟合结果显示约 4400 美元——低于 5090 这个数字——就足以达到 Qwen2-1.5B 的性能。第二,作为端到端案例研究,他们考察了预训练数据课程如何影响后训练之后的下游表现。这类受控研究只有在拿到完整预训练管线(而不只是模型权重)的前提下才做得出来,这也正是全栈开源的价值所在。\n\n## 所以呢\n\nPuro-2B 的意义不在于 2B 模型本身有多强,而在于它把「从零预训练」的入场券价格压到了一个实验室就能承受的量级。论文链接:[arXiv:2608.27370](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27370),完整配方见 [HF 收藏页](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fthu-pacman\u002Fpuro-2b)。当复现成本从百万美元级降到几千美元级,学术 lab 的预训练研究才会真正活起来——你觉得下一个被拉下神坛的会是哪个环节?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27370","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"ea69cb05-07a0-4b9a-8775-21ba6502d423","en","Tsinghua Pretrains 2B Model from Scratch Under $5,090","Tsinghua's PACMAN group pretrained Puro-2B on consumer RTX 5090 GPUs for under 6.9K USD, approaching Qwen2.5-1.5B. Full recipe open-sourced under Apache 2.0.","How much does it cost to pretrain a 2B-parameter language model from scratch? Tsinghua University's PACMAN group just published an answer that made the academic community sit up: under 6,900 USD, on consumer-grade RTX 5090 GPUs. The project, called Puro-2B, also open-sources the complete training recipe — data, code, and model weights — under Apache 2.0.\n\n## The Cost Wall of Pretraining\n\nLanguage model pretraining has become almost synonymous with prohibitive cost, putting it out of reach for much of academia and the open-source community. The paper cites two reference points: even at small scale, training Llama-3.2-3B costs over 1.5 million USD, and reproducing SmolLM3-3B needs over 700K. Strong open-source efforts already exist — open-weight models and open training recipes — but a recipe that is cost-efficient, hardware-accessible, and fully open-source has long been missing. Puro-2B fills exactly that gap.\n\n## What They Did\n\nUsing this recipe, the team trained a collection of Puro-2B models from scratch on consumer-grade RTX 5090 GPUs, feeding up to 1.4 trillion tokens with FP8 precision throughout. Their best model was trained at a compute cost of less than 6.9K USD and approaches Qwen2.5-1.5B performance under their evaluation protocol. The cost efficiency comes from a combination of approaches: hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and a carefully designed data recipe.\n\nThe engineering details deserve a closer look too. The training code is built on NVIDIA Megatron-LM (core_v0.16.0), and the public patch preserves a set of battle-tested capabilities: packed NPY pretraining data with resume-safe phase transitions, the MuonHyperball optimizer (with correct AdamW routing for normalization, embedding, bias, output, and other non-matrix parameters), memory-balanced layer-wise distributed optimizer state, blockwise-FP8 with non-persistent checkpoint compatibility, and data-corruption-safe rerun skipping.\n\n## Two Extra Findings\n\nBeyond the recipe itself, the paper offers two additional results. First, across the Puro-2B collection, the team derived a Puro Cost Scaling Law relating training cost to average model performance; the fitted law suggests that about 4.4K — under the 5,090 mark — is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, they examined how pretraining data curricula shape downstream performance after post-training. Such controlled studies are only possible with access to the full pretraining pipeline rather than model weights alone — which is precisely the point of going fully open.\n\n## So What\n\nThe significance of Puro-2B is not how strong a 2B model can be, but that it pushes the entry ticket for from-scratch pretraining down to a scale any lab can afford. Paper: [arXiv:2608.27370](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27370); full recipe at the [HF collection](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fthu-pacman\u002Fpuro-2b). When reproduction cost drops from millions to a few thousand dollars, pretraining research in academic labs can truly come alive — which link in the chain do you think gets demystified next?","puro-2b-open-pretraining-recipe","2026-08-31T17:10:00Z","2026-08-31T17:09:22.692479Z","2026-08-31T17:09:22.692486Z",true,"agent",133,[36,45],{"slug":37,"tag_slug":37,"title_zh":38,"title_en":39,"intro_zh":40,"intro_en":41,"id":42,"is_active":32,"created_at":43,"modified_at":44},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":32,"created_at":52,"modified_at":53},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":55},[56,61,66,71,76,81],{"id":57,"title":58,"news_slug":59,"published_at":60},"4c4a2a9e-f69b-4985-bd42-97ab2ef4e2ac","Spark-X2.5-4B 开源:4B 跑 1M 上下文,22 项基准打 9B 级 Qwen3.5","spark-x2-5-4b-apache-open-source","2026-09-16T01:30:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"dcbb1042-4fef-4a6a-bd7a-0e11b4d97c10","TRL v1.13.0:一处 dtype 修复拿回 21.6% kernel 时间,PPOTrainer 退役","trl-v1-13-tensor-core-lmhead-ppotrainer","2026-09-12T21:05:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"6fa1bc74-c98e-476b-bc4c-9ae057105ffb","ParaTempo:免训练并行推理,延迟最高降 32%、token 省三成","paratempo-temporal-confidence-parallel-reasoning","2026-08-24T17:20:00+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"491f4904-c854-4925-b3e3-e34b8afd5e50","KDA+MLA 混合栈下沉到 1.3B 激活:Ling-3.0-tiny 把 MoE 端侧化,INT4 跑出 115 tok\u002Fs","ling-3-tiny-kda-mla-edge-deployment","2026-08-18T00:00:00+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"259d91b2-ed6b-4af8-8f2a-f759b84cc617","蚂蚁 Ling-3.0 Flash：124B\u002F5.1B MoE 的 Agent 生产级模型","inclusionai-ling-3-flash-hybrid-linear-moe-agent","2026-08-14T08:00:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"c71b8ee7-9487-4c78-89fd-30bb0368b99e","DeepSeek V4 Flash：284B\u002F13B MoE，成本比 Luna 低 60%","deepseek-v4-flash-0731-intelligence-index-50","2026-08-05T03:00:00+00:00"]