[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-cola2-hierarchical-language-modeling":3,"topics-all":38,"news-related-aebb62ce-1c13-45ce-a83c-228cc2a2cc50":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"aebb62ce-1c13-45ce-a83c-228cc2a2cc50","字节Seed:语义压进连续潜变量,346B追近Llama-3","字节 Seed 的 Seedance 团队发布层级语言生成框架 Cola2:文本先被压缩成连续语义潜变量,再由掩码扩散解码器生成 token。346B 源 token 的未收敛版本八任务平均 60.10,两项基准超过 Llama-3-8B,论文称 token 预算差 43 倍;8B 模型计划数周内开源。","逐 token 自回归是不是语言模型的唯一解?字节 Seed 的 Seedance 团队联合港大、北大、人大的研究者刚在 arXiv 放出 47 页长文 Cola2,把另一条路线做了系统的工程化拆解:先把文本压缩成连续语义潜变量,再让扩散解码器去\"填词\"。论文地址:https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31305。\n\n## 两级流水线怎么搭\n\nCola2 的核心是\"压缩-生成\"分工。第一阶段训练压缩编码器:文本被动态切分成内容感知的语义段,潜变量按段分配,压缩率由一个无辅助损失的负反馈控制器稳定在目标附近,避免与表示学习互相打架。第二阶段引入约 4B 参数的潜空间 DiT:用 Flow Matching 拟合潜变量分布,其梯度经 stop-gradient 与编码器解耦;token 侧的预测梯度则继续穿过潜变量更新编码器,驱动连续表示与离散生成协同演化。第三阶段单独续训 DiT。\n\n解码器是块因果掩码扩散模型(MDM),训练时按丢弃、加噪、干净 0.4\u002F0.4\u002F0.2 的比例混合三种潜变量条件;推理时 CFG 负责在\"忠实重建语义\"与\"自主生成剩余细节\"之间调平衡。消融显示打乱潜变量会让精度大跌 12.2 个点,证明解码器确实在用对齐的语义内容。整条推理链路是\"DiT 提议潜变量块 → 解码器实现成 token → 固定段布局下重编码回填\",DiT 去噪约 8 步就饱和。\n\n## 346B token 的成绩单\n\n最大一次预训练从 Qwen3-4B-Base 初始化因果解码器,配合约 4B 的潜空间 DiT,只用 346.03B 源 token、且明确未收敛,就在八项基准上拿到 60.10 的平均分,距 Llama-3-8B(66.25,报告预算 15T+ token)仅 6.15 分;HellaSwag(41.75 对 39.21)和 StoryCloze(80.42 对 79.49)两项反超。论文强调 token 预算相差 43 倍以上。项目页给出完整对照:Qwen2.5-7B(18T)平均 75.08,Qwen3-8B-Base(约 36T)75.87,LLaDA-8B-Base(2.3T)72.11。一作还在 Hugging Face 社区宣布,8B 模型计划数周内开源,项目页 Code 与 Model 目前都挂着 Soon。\n\n## 该泼的冷水\n\n第一,43 倍效率是论文口径,而且解码器站在 Qwen3-4B-Base 的肩膀上——论文自己也把成绩归因于\"Qwen 初始化解码器生成能力的有效迁移\",这不是从零训练的 346B 神话。第二,压缩在理论上砍掉潜空间先验的线性与注意力 FLOPs,但端到端生成速度还取决于解码器、CFG 分支与硬件利用率,作者明说需要专用 kernel、缓存管理和并行调度的基建协同设计才能兑现。第三,消融数字很诚实:生成监督把精度从 15.7% 拉到 26.7%,编码器-解码器共训带来 5.9 个点,但压缩版 Cola2-MDM 收敛更慢,要到约 150B token 后才持续缩小与 LLaDA 的差距。\n\n所以呢:Cola2 证明的是\"连续潜变量 + 离散生成\"这条路能扩、能打,不是它已经赢了。范式竞赛的裁判从来是每 token 成本——等 8B 权重落地、第三方复现跑分,才是这条路线的真分水岭。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31305","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"73b0b009-461c-4c5a-b5cb-c5c0554e11bc","en","ByteDance Seed's Cola2 Nears Llama-3-8B With 346B Tokens","Cola2 compresses text into semantic latents, decoded by masked diffusion; 346B tokens beat Llama-3-8B on two of eight tasks. 8B model due soon.","Is next-token autoregression the only way to build a language model? Researchers from ByteDance Seed's Seedance team, with the University of Hong Kong, Peking University and Renmin University of China, just posted a 47-page arXiv paper, Cola2, that gives the alternative route its most systematic engineering treatment yet: compress text into continuous semantic latents first, then let a diffusion decoder do the writing. Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31305.\n\n## A two-level pipeline\n\nCola2's core is a compression-generation division of labor. Stage one trains the compression encoder: text is dynamically segmented into content-aware semantic spans, latents are allocated per span, and the compression rate is held near target by an auxiliary-loss-free negative-feedback controller, so it never fights representation learning. Stage two introduces a roughly 4B-parameter latent DiT: Flow Matching fits the latent distribution, with its gradient decoupled from the encoder via stop-gradient, while token-prediction gradients keep flowing through the latents to update the encoder, so continuous representation and discrete generation co-evolve. Stage three continues training the DiT alone.\n\nThe decoder is a block-causal masked diffusion model (MDM) trained on a mixture of dropped, noisy and clean latent conditions at a 0.4\u002F0.4\u002F0.2 ratio; at inference, classifier-free guidance tunes the balance between faithful semantic reconstruction and autonomous generation. Ablations show shuffling the latents costs 12.2 points, evidence the decoder really uses the aligned semantics. The inference loop is propose, decode, re-encode: the DiT proposes a latent block, the decoder realizes tokens, and the generated text is re-encoded under a fixed span layout. DiT denoising saturates at about eight steps.\n\n## The 346B-token report card\n\nThe largest pretraining run initializes the causal decoder from Qwen3-4B-Base plus a roughly 4B latent DiT. Using only 346.03B source tokens, and explicitly unconverged, it reaches a 60.10 average over eight benchmarks, 6.15 points behind Llama-3-8B (66.25, reported budget 15T+ tokens), while surpassing it on HellaSwag (41.75 vs 39.21) and StoryCloze (80.42 vs 79.49). The paper stresses the token budgets differ by more than 43 times. The project page's full table: Qwen2.5-7B (18T) averages 75.08, Qwen3-8B-Base (about 36T) 75.87, LLaDA-8B-Base (2.3T) 72.11. The first author also announced on Hugging Face that an 8B model will be open-sourced in the coming weeks; the project page's Code and Model links still read \"Soon\".\n\n## The cold water\n\nFirst, the 43x efficiency figure is the paper's own framing, and the decoder stands on Qwen3-4B-Base's shoulders — the authors themselves attribute results to effective transfer of the Qwen-initialized causal decoder's generation capability. This is not a from-scratch 346B miracle. Second, compression cuts the latent prior's linear and attention FLOPs in theory, but end-to-end speed still depends on the decoder, CFG branches and hardware utilization; the authors explicitly call for infrastructure co-design — specialized kernels, cache management, parallel scheduling — to turn theoretical savings into real speedups. Third, the ablations are honest: generative supervision lifts accuracy from 15.7% to 26.7%, encoder-decoder co-training adds 5.9 points, but the compressed Cola2-MDM converges more slowly and only starts closing the gap with LLaDA after about 150B tokens.\n\nSo what: Cola2 proves the continuous-latent-plus-discrete-generation route scales and competes, not that it has won. The judge of paradigm contests has always been cost per token; the real watershed arrives when the 8B weights land and third parties reproduce the numbers.","cola2-hierarchical-language-modeling","2026-10-23T19:08:22Z","2026-10-23T19:08:26.681138Z","2026-10-23T19:08:26.681151Z",true,"agent",26,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"4b73ca86-414e-454b-b2bf-df0d60e18d65","扩散LLM量化:T2Quant按去噪时刻分配精度","t2quant-task-aware-diffusion-llm-quantization","2026-10-24T01:24:29+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"02c96622-1409-4867-9f25-772936edc6cd","滑动窗口反超SnapKV:NVIDIA重审投机解码KV压缩","rift-draft-kv-cache-compression","2026-10-23T15:20:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"a046b158-396f-4db9-a8d3-22d8c193a166","清华把KV缓存当神经网络训练:32倍压缩准确率96.7%","latent-compaction-test-time-kv","2026-10-15T17:08:40+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a820bb5a-ac48-4aab-8d8b-34da340900c1","REMORY 让 LLM 用 5% 上下文继续跑 Agent","remory-agent-memory","2026-10-11T09:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"dab2e487-cfba-4d75-a93f-eff133476348","剪枝校准对不上解码分布?西湖蚂蚁让稀疏解码提速48%","sparsedecoding-decoding-aware-pruning","2026-10-10T19:09:23+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"a78b61ba-6899-4b4f-86b0-8d6c1753308b","亚马逊 ALoDLM:扩散语言模型学会按难度分配算力","alodlm-adaptive-looped-diffusion","2026-10-07T13:15:53+00:00"]