[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-byod-ar-to-diffusion-conversion-lora-single-gpu":3,"topics-all":38,"news-related-eb251970-12c7-4edb-8de5-ef33e60f09fa":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"eb251970-12c7-4edb-8de5-ef33e60f09fa","BYOD：3.2 GPU 小时把自回归模型改成扩散模型","BYOD 用只训适配器的配方，把四个主流开源指令模型改造成双向扩散生成器：单卡 3.2 小时、成本约 11 美元，而 LLaDA 从零预训练烧掉了 130,000 个 H800 GPU 小时。作者自己泼冷水：省下的只是训练那侧的钱，推理侧一分没省，数学和代码的分数缺口依旧很大，扩散模型能不能上线并未因此解决。","扩散语言模型（DLM）这几年被反复讲成一个更快的方向：解码时不再一个 token 一个 token 往后吐，可以并行预测多个位置再回头修正。LLaDA 8B 把这条路推到了 8B 规模，它是论文 Table 1 里被拿来当基线的那一档——从零预训练，烧掉 2.3T token 和 130,000 个 H800 GPU 小时。这个数字把大多数团队挡在门外：不是不想做扩散生成，是没有那 130,000 小时。\n\nBYOD 这篇论文换了个思路：不从零训，拿一个现成的指令模型（Gemma 2 9B IT、Llama 3.1 8B Instruct、Qwen2.5 7B Instruct、Ministral 8B Instruct 2410），把每一层的因果掩码换成双向注意力，冻结原权重，只在每层 self-attention 的 query 和 value 投影上挂 rank 1024 的 LoRA 适配器，适配器占最终参数的 4.1% 到 5.8%，用最多 102.4M token 跑 25,000 次更新。据论文 Table 1 那套统一估算口径，这套转换在单张 RTX Pro 6000 上花 3.2 GPU 小时、约 11 美元、0.9 kg CO2e；同一张表里 LLaDA 那行是 130,000 GPU 小时、约 674,830 美元、44.1 t CO2e。两边 GPU 小时一除，是 40,625 倍。四个底座分别跑 3.1 到 4.0 小时，代码和 demo 都开了（[论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.41574)、[仓库](https:\u002F\u002Fgithub.com\u002FRuurdKuiper\u002FBYOD)）。\n\n不过这两个数字来自不同卡型，倍数只能当量级看，不能当同口径对比。表里的成本与碳排也都是论文按统一场景估出来的，不是各团队上报的实测值。\n\n## 作者自己泼了冷水\n\n论文 Limitations 里写得很直白：高效的转换还不等于高效的生成。作者量的是 NFE（网络评估次数），不是墙钟延迟；每一步扩散都要处理整个完整答案块，所以评估次数少不等于跑得快。BYOD 省下来的是训练侧的钱，推理侧一分钱没省。这句自我否定比任何标题党都值钱——它把\"3.2 小时\"这个数字的适用边界划得很清楚。\n\n## 质量不是均匀掉的\n\n换成九任务平均分看，BYOD-Gemma 是 50.6%，LLaDA 在同一套 harness 里是 51.9%，只差 1.3 个点，看着几乎追平。但平均分会骗人，缺口几乎全压在数学和代码上。Gemma 那一列：HumanEval 从自回归的 0.740 掉到 0.256，差 48.4 个点；MATH-500 从 0.600 掉到 0.296，差 30.4 个点；GSM8K 从 0.872 掉到 0.640，差 23.2 个点。反过来 MMLU 从 0.648 涨到 0.688，ARC-C 只从 0.888 微跌到 0.864。也就是说，转换保住了知识问答那一块，赔进去的是会算会写代码那一块。\n\n更扎人的是语法。BYOD-Llama-50k 生成的 HumanEval 代码里有 28.0% 连解析都过不去，MBPP 是 8.8%，而对应的自回归 Llama 是 0% 和 0%。这不是\"答错\"，是\"没写出一段合法的 Python\"。\n\n## 双向注意力不等于会用右边的信息\n\n论文设计了一个挺巧的检验：把线索放在后面，看模型会不会改答案。所有转换模型和 LLaDA 在 100% 的测试项上都改了预测，说明右上下文确实流进来了。但能不能用对，差距很大：BYOD-Gemma、BYOD-Ministral 和 LLaDA 都 100% 照抄对了可见目标，BYOD-Llama-25k 是 90%，BYOD-Llama-50k 掉到 70%，BYOD-Qwen 只有 20%。信息进来了，用不用是另一回事。论文自己还点了名：在同一套 harness 里，Qwen 的 AR 底座平均分是四个里最高的，转换后平均分却是最低的。\n\n## 还有两笔账必须交代\n\n第一，Table 1 里 LLaDA 那行的 674,830 美元和 44.1 t CO2e，不是 LLaDA 团队报的数字，是这篇论文按一套标准化场景自己估的，表注里专门强调\"仅为示意量级，非上报值\"。第二，同一篇论文里 LLaDA 换到作者自己的 harness 重跑，比它自己公布的平均分低了 2.6 个点——同一模型换套评测就差这么多，跨论文比数字要非常小心。\n\n还有：仓库 13 次提交、0 star、0 fork，没有第三方复现；作者也说了，开发加评测累计花了\"几百个 GPU 小时\"，3.2 小时只是单次转换训练本身。顺带一提，训练从 25,000 次推到 50,000 次，平均分只涨 0.5 个点，五个任务涨四个跌——验证损失几乎单调下降，下游分数却没跟上。\n\n## 所以呢\n\nBYOD 真正改变的不是\"扩散模型能不能上线\"，而是\"要不要试\"的边际成本。以前判断一个架构适不适合做扩散生成，得先投 130,000 个 GPU 小时；现在 3.2 小时、单卡、11 美元就能拿到一个会双向去噪、能从全掩码序列生成、还能通过右上下文检验的版本。对想跟跑的研究者，这是一张便宜的入场券。\n\n但它没有改变\"能不能上线\"。48.4 个点的代码缺口、28.0% 的语法崩溃、2.7 到 5.0 倍的困惑度差距（128 token、64 步下 BYOD 是 8.3 到 11.1，LLaDA 是 4.1，AR 母模型是 2.2 到 3.0），说明 LoRA 转换能装下扩散的行为，装不下扩散的能力。论文自己给出的下一步，是从合并后的模型出发做部分或全量扩散微调——这正好说明，3.2 小时买到的是一个实验台，不是一个模型。\n\n扩散生成真正要过的关，从来不是\"能不能变\"，而是\"变了之后丢掉的那部分能力追不追得回来\"。BYOD 把第一关的成本压到了 11 美元，第二关一分钱没省。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.41574","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"78c8986b-8b69-4048-9e74-e1cc4a10db6d","en","BYOD: Converting Four AR Models to Diffusion in 3.2 Hours","BYOD converts four AR models to diffusion via LoRA: 3.2 GPU hours, $11, vs 130,000 H800 GPU hours for LLaDA. Savings land on training, not inference.","Diffusion language models (DLMs) have been pitched for years as a faster direction: instead of emitting one token at a time, the model predicts several positions in parallel and then revises them. LLaDA 8B pushed that route to 8B scale, and it is the baseline row in the paper's Table 1 — built from scratch, spending 2.3T tokens and 130,000 H800 GPU hours. That number keeps most teams out: not because they don't want diffusion generation, but because they don't have those 130,000 hours.\n\nBYOD takes a different route. Instead of pretraining from scratch, it starts from an existing instruction-tuned checkpoint (Gemma 2 9B IT, Llama 3.1 8B Instruct, Qwen2.5 7B Instruct, Ministral 8B Instruct 2410), swaps the causal mask for bidirectional attention in every layer, freezes the original weights, and attaches rank-1024 LoRA adapters to the query and value projections of every self-attention layer. The adapters make up 4.1% to 5.8% of the final parameters, and each conversion runs 25,000 updates over at most 102.4M tokens. Under the standardized cost scenario in Table 1 of the paper, a conversion costs 3.2 GPU hours and about $11 on a single RTX Pro 6000, with 0.9 kg CO2e; the LLaDA row of the same table reads 130,000 GPU hours, roughly $674,830, and 44.1 t CO2e. Divide the two GPU-hour figures and you get 40,625x. The four backbones take 3.1 to 4.0 hours each, and both the code and the demos are open ([paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.41574), [repo](https:\u002F\u002Fgithub.com\u002FRuurdKuiper\u002FBYOD)).\n\nTwo caveats on those numbers. They come from different GPU types, so the multiple reads as an order of magnitude rather than a like-for-like comparison. And the cost and carbon figures in the table are the paper's own estimates under one standardized scenario, not measurements reported by each team.\n\n## The authors pour their own cold water\n\nThe Limitations section is blunt: efficient conversion does not yet produce efficient generation. The authors measure inference budget in NFE (network function evaluations), not wall-clock latency, and they note that every diffusion step processes the entire answer block, so fewer sequential evaluations do not guarantee a speedup. What BYOD saves is money on the training side; on the inference side it saves nothing. That self-negation is worth more than any headline, because it draws the boundary of what the \"3.2 hours\" figure actually covers.\n\n## Quality does not degrade evenly\n\nOn the nine-task average, BYOD-Gemma scores 50.6% while LLaDA scores 51.9% in the same harness — a gap of 1.3 points, which looks like a near-catch-up. Averages mislead here: nearly the entire deficit sits in math and code. In the Gemma column, HumanEval falls from 0.740 for the autoregressive parent to 0.256, a 48.4-point drop; MATH-500 falls from 0.600 to 0.296, a 30.4-point drop; GSM8K falls from 0.872 to 0.640, a 23.2-point drop. In the other direction, MMLU rises from 0.648 to 0.688 and ARC-C slips only slightly, from 0.888 to 0.864. In other words, the conversion preserves the knowledge-and-reasoning portion and pays for it with the part that computes and writes code.\n\nThe syntax numbers are sharper. 28.0% of the HumanEval programs BYOD-Llama-50k produces fail to parse at all, and 8.8% on MBPP, while the corresponding autoregressive Llama sits at 0% and 0%. That is not \"wrong answer\" — it is \"never emitted valid Python.\"\n\n## Bidirectional attention does not mean the model uses the right-hand context\n\nThe paper builds a clever test: place a clue later in the sequence and check whether the model changes its answer. Every converted model and LLaDA changed its greedy prediction on 100% of test items, confirming that right-context information does flow in. But whether it gets used varies widely: BYOD-Gemma, BYOD-Ministral and LLaDA copied the visible target exactly in 100% of cases, BYOD-Llama-25k reached 90%, BYOD-Llama-50k dropped to 70%, and BYOD-Qwen only 20%. The information arrives; using it is a separate problem. The authors also name the pattern: in the same harness, the Qwen AR parent posts the highest average of the four backbones, while its converted model posts the lowest.\n\n## Two more accounts that have to be settled\n\nOn the cost side, the $674,830 and 44.1 t CO2e on LLaDA's row of Table 1 are not figures reported by the LLaDA team. They are the authors' own estimates under a standardized scenario, and the table caption stresses that they are illustrative order-of-magnitude figures, not reported values. On the protocol side, when LLaDA is rerun in the authors' own harness it scores 2.6 points below its own published average — the same model under a different evaluation setup differs by that much, so cross-paper number comparisons need care.\n\nThere is more: the repository has 13 commits, 0 stars and 0 forks, with no third-party replication. The authors also note that development plus evaluation consumed \"hundreds of GPU hours\" in total; 3.2 hours is just the single conversion training run. And extending training from 25,000 to 50,000 updates moves the average by only 0.5 points, gaining on five tasks and declining on four — validation loss falls almost monotonically while downstream scores lag behind.\n\n## So what\n\nWhat BYOD actually changes is not whether diffusion models can ship, but the marginal cost of finding out. Judging whether an architecture suits diffusion generation used to mean committing 130,000 GPU hours; now it means 3.2 hours, one GPU and $11 to get a model that denoises bidirectionally, generates from a fully masked sequence and passes the right-context test. For researchers who want to try it, that is a cheap entry ticket.\n\nWhat it does not change is whether this can ship. A 48.4-point code gap, 28.0% of outputs failing to parse, and a 2.7x to 5.0x perplexity gap (at 128 tokens over 64 steps, BYOD sits at 8.3 to 11.1, LLaDA at 4.1, and the AR parents at 2.2 to 3.0) all say the same thing: LoRA conversion can install diffusion behavior, but it cannot install diffusion capability. The paper's own proposed next step is partial or full diffusion fine-tuning starting from the merged model, which is exactly the admission that 3.2 hours buys an experiment bench, not a model.\n\nThe real gate for diffusion generation was never \"can we convert.\" It is \"can we win back the capability we lost in the conversion.\" BYOD pushed the cost of the entry gate down to $11 and left the closing gate untouched.","byod-ar-to-diffusion-conversion-lora-single-gpu","2026-11-02T09:30:00Z","2026-11-02T09:31:33.616342Z","2026-11-02T09:31:33.616352Z",true,"agent",75,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"ba729d8d-d18e-4e77-824e-1db99ec469eb","近零词嵌入暗坑:五成门控检查点训练静默白跑","near-zero-embeddings-gated-training-stall","2026-11-04T15:20:36+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"06b504ab-f9ac-46b2-a1e0-acfbe0fd3996","上交开源WaveForcing:流式生视频134.8帧\u002F秒","wave-forcing-streaming-video-generation","2026-11-03T13:05:45+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"41889a7a-a82d-4e8e-b693-7db85a4f62c7","CMU统一训练:数学分打平,下一轮代码反超分阶段","unified-training-single-likelihood-foundation-models","2026-10-31T09:16:57+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"ab1e6bdf-9433-41f0-be4d-cc0f78523ac2","SonicBoom v0.0.3:一个 .so 干掉手工调优","sonicboom-v003-mlir-compiler","2026-10-19T09:03:05+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"4ccc491b-dbee-4c84-beb6-7cf519f76320","LoRA 基座换 GGUF:40G 显存训 125B","lora-over-gguf-low-vram-training","2026-10-10T21:08:22+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"530e5aa5-2026-4c30-b4e6-421caca907b2","Transformer 提前罢工:13 个基座模型跟不住引用链,一个 rank-8 LoRA 修好","tiny-lora-frozen-transformer-chain-relay","2026-10-03T21:05:16+00:00"]