[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tencent-mc-sparse-h3-diffusion-1-8x":3,"topics-all":38,"news-related-bbcda72c-dc57-4b0f-acfe-103b8e80ff8e":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"bbcda72c-dc57-4b0f-acfe-103b8e80ff8e","腾讯混元 MC-Sparse:无重训 1.8 倍稀疏加速","腾讯混元把扩散 Transformer 的稀疏注意力问题拆成 3 块,training-free 方案在 MiniMax-H3-Base 上做到 1.80 倍去噪加速、3D 资产 2.32 倍加速,画质几乎无损。","扩散 Transformer 跑视频和高分辨率 3D 资产生成已经很贵,稀疏注意力本来是降本的标准路径——只是各家现成方案在高稀疏率下都会损伤画质。腾讯混元 10 月 5 日挂在 arXiv 的 MC-Sparse (arXiv 2610.06801) 把这件事重新拆了一遍:在 MiniMax-H3-Base 上做到 1.80 倍去噪加速,在 3D 资产生成上做到 2.32 倍加速,画质损失几乎不可见;更关键的是,这是一套完全 training-free 的方案,不需要重新训练任何模型。\n\n## 现有稀疏注意力方案的三个根问题\n\n作者先用一组对照实验把「为什么现有的稀疏注意力方案在高稀疏率下画质会崩」拆成了三个来源:\n\n- **token grouping 带来的结构约束**:为了适配 GPU 的 tile 大小,大多数方案会强制把多个 query 或 KV 打包成一组,这等于人为增加了一层粒度,模型没法在这个粒度上表达「真正重要」的 token;\n- **交互选择不准**:基于近似分数或滑动窗口去挑 KV,经常漏掉真正参与贡献的 token,而且挑错之后没有办法在后续步骤里纠正;\n- **被丢弃 token 的注意力贡献消失**:一旦 KV 被丢掉,它在 dense attention 里贡献的那部分分数就彻底没了,后面所有 denoise step 都补不回来。\n\n这三条里只要任何一条没处理好,画面就会出现闪烁、细节糊掉、人物面部崩坏这些肉眼可见的问题。\n\n## MC-Sparse 的做法:分块选 token + 复用元数据\n\nMC-Sparse 的核心叫 Meta-Cached Sparse Attention,做了三件事来解决上面的三个问题:\n\n- **tile 对齐的 query 分组**:不是粗暴地把 query 切块,而是把相似的 query 用 Fast PDDP 算法聚成一组,组内共享 GPU 的调度开销——解决了「grouping 的结构约束」;\n- **用真实的 attention 概率选 KV**:在「anchor step」(校准步骤)里跑一次 dense attention,拿到的真实概率分布来挑每个 query 真正需要的 KV token,而不是用近似分数猜——解决了「交互选择不准」;\n- **把 dense 和 sparse 的输出残差缓存下来**:一次性算好 dense baseline 和 sparse 输出之间的差,缓存进 metadata,后面所有 denoise step 直接复用,再叠加 fresh 的 Q\u002FK\u002FV——解决了「被丢弃 token 的贡献消失」。\n\n换句话说,这套方案不是「少算点 attention」,而是「少算点 attention,但把算过的那点信息压榨到极致」。\n\n## 数据:1.80 倍 \u002F 2.32 倍,无画质损失\n\n论文里给出的对照数字(基于 MiniMax-H3-Base 和 3D 资产生成模型):\n\n- 视频生成(MiniMax-H3-Base):**1.80× denoising speedup**,与 dense attention 相比画质损失可忽略;\n- 3D 资产生成:**2.32× speedup**,几何细节保留明显优于现有稀疏注意力 baseline。\n\n仓库 `dodododddo\u002Fmcsparse` 已经开源,Apache-2.0 协议,内含完整的 kernel 实现和 MiniMax-H3 集成。Setup 章节明确写了**目前只支持 NVIDIA Hopper 架构 GPU**,需要 CUDA-enabled PyTorch + FlashAttention-3 的 `flash_attn_interface`。\n\n## 部署与生态\n\nGitHub README 给出了几种典型 GPU 布局的脚本:\n\n- 单卡 5 秒 \u002F 15 秒短视频生成(`run_1gpu_5s.sh` \u002F `run_1gpu_15s.sh`);\n- 7+1 卡的服务化布局(1 张卡常驻 text encoder,其余 7 张用 Ulysses context parallelism 做 denoise);\n- 8 卡共享布局(encoder 用完回到 CPU,腾出显存给 denoise)。\n\n调度的关键开关包括 `INDEX`、`OUT_DIR`、`GPU` \u002F `NPROC`、`TOPK` \u002F `TOPKS`、`CALIBRATE`——可以在不重训模型的前提下,根据硬件配置灵活调稀疏率。\n\n## 实用边界与 TODO\n\n作者在 README 的 TODO 部分给得很坦白:\n\n- Fast PDDP 的 query grouping 速度还要进一步优化;\n- whitening 支持还要在 H3 集成里补完并验证。\n\n实际可落地的功能已经覆盖了 MiniMax-H3 的端到端采样流程——`mcsparse info`、`mcsparse examples`、`mcsparse run` 几条命令就能跑通视频和 3D 资产两条管线。Benchmark(`tests\u002Fbench_token_sparse.py`、`tests\u002Fbench_pca_group.py`)和纯逻辑测试(`tests\u002Ftest_token_sparse_logic.py`)都已附带,不用 PyTorch\u002FCUDA 也能跑调度逻辑那一层。\n\n## 评论:Hunyuan 把「稀疏注意力」的工程化门槛砍掉了\n\n稀疏注意力在视频扩散领域已经被反复研究过,但真正能在生产里挑大梁的方案一直很少——大多数论文要么改了一行 attention 的实现就要重新训练,要么稀疏率上去了画质就崩。MC-Sparse 给出了第三条路:**不重训、用真实概率选 token、把残差缓存复用**。\n\n这恰好踩到了 MiniMax-H3、3D 资产这类长序列扩散模型当下最痛的点——去噪 step 数本身已经压不下去,只能在每一步里省 FLOPs。1.80 倍 \u002F 2.32 倍的加速意味着同一张卡能多跑一档分辨率,或多撑一倍 batch。\n\n唯一遗憾:**目前只支持 Hopper**。这对 A100 \u002F A800 \u002F L40 \u002F 国产卡集群是个明确的不友好信号——意味着要在新架构上重写 CuTeDSL kernel。考虑到腾讯自己也是 NV 用户,这很可能只是「先支持自有用卡」的策略;但对其他厂商的部署方来说,等社区 PR 还是看 Tencent 后续动作,是接下来值得跟进的点。\n\n代码与论文:\n- 论文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.06801\n- GitHub:https:\u002F\u002Fgithub.com\u002Fdodododddo\u002Fmcsparse (Apache-2.0)\n","https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2610.06801","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",{"id":19,"name":20,"slug":20,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d44ac9ab-2054-46db-b72b-c35686ae070f","en","MC-Sparse: Training-Free 1.8x Sparse Attention Speedup","A training-free sparse attention framework from Tencent Hunyuan hits 1.80x denoising speedup on MiniMax-H3-Base and 2.32x on 3D asset generation.","Diffusion transformers driving video and high-resolution 3D asset generation are notoriously expensive, and sparse attention is the standard lever for cutting cost — but existing approaches either degrade quality at high sparsity ratios or demand retraining. A 5 October arXiv paper from Tencent Hunyuan (arXiv 2610.06801) proposes a third path: MC-Sparse, a training-free meta-cached sparse attention framework that hits 1.80x denoising speedup on MiniMax-H3-Base and 2.32x on 3D asset generation, with no visible quality loss.\n\n## Three root causes the paper pins down\n\nThe authors run controlled oracle comparisons and trace existing methods' high-sparsity degradation to three sources:\n\n- Token grouping constraints — Most sparse-attention implementations force queries or KVs into fixed-size GPU tiles. This adds a granularity the model cannot express in terms of which tokens actually matter;\n- Inaccurate interaction selection — Picking KV tokens via approximate scores or sliding windows consistently misses the ones that contribute real attention mass, with no way to recover later;\n- Lost contributions from discarded tokens — Once a KV is dropped, its share of the dense-attention output is gone for every subsequent denoising step.\n\nAny of these three being mishandled shows up as flickering, blurred details, or facial artifacts in the generated frames.\n\n## What MC-Sparse actually does\n\nThe framework, Meta-Cached Sparse Attention, attacks each cause:\n\n- Tile-aligned query groups — Instead of chopping queries into fixed tiles, MC-Sparse uses a Fast PDDP algorithm to cluster similar queries into groups that share GPU scheduling overhead;\n- Exact attention mass for KV selection — At anchor steps the framework actually runs dense attention, then uses the real probability distribution to pick the KV tokens each query truly needs;\n- Cached residuals between dense and sparse outputs — The difference between the dense baseline and the sparse output is computed once, cached as metadata, and reused across all subsequent denoising steps where fresh Q\u002FK\u002FV are computed normally.\n\nThe headline is not compute less attention but compute less attention while squeezing every last bit of value out of what you did compute.\n\n## Numbers, deployment, and the Hopper ceiling\n\nThe reported numbers come straight from the paper and the project page:\n\n- Video generation on MiniMax-H3-Base: 1.80x denoising speedup, negligible quality loss vs dense attention;\n- 3D asset generation: 2.32x speedup, visibly better geometry preservation than existing sparse baselines.\n\nCode lives at dodododddo\u002Fmcsparse under Apache-2.0, with the kernel implementation and a MiniMax-H3 integration included. The README is explicit about one major constraint: only NVIDIA Hopper GPUs are supported. Setup needs a CUDA-enabled PyTorch plus a FlashAttention-3 build exposing flash_attn_interface. The repo ships with several layout scripts:\n\n- Single-GPU 5s \u002F 15s video generation (run_1gpu_5s.sh \u002F run_1gpu_15s.sh);\n- 7+1 serving-oriented layout (one GPU keeps the text encoder resident while the rest denoise with Ulysses context parallelism);\n- 8-GPU shared layout (encoder returns to CPU after encoding, freeing GPU memory for denoise).\n\nSwitches worth knowing: INDEX, OUT_DIR, GPU or NPROC, TOPK or TOPKS, CALIBRATE.\n\n## An honest TODO and what is actually shipped\n\nThe README TODO list is refreshingly concrete: further optimize Fast PDDP query-grouping speed; expose and validate whitening support in the H3 integration and CLI. What is shipped covers the full MiniMax-H3 sampling pipeline — mcsparse info, mcsparse examples, and mcsparse run drive both video and 3D asset generation end to end. Benchmarks (tests\u002Fbench_token_sparse.py, tests\u002Fbench_pca_group.py) and a logic-only test suite (tests\u002Ftest_token_sparse_logic.py) are included, the latter running without PyTorch or CUDA.\n\n## Why this matters for diffusion deployment\n\nSparse attention for video diffusion has been studied for years, but production-grade options have been thin — most papers either change one line of attention and demand a retrain, or push sparsity up until quality collapses. MC-Sparse threads a third route: no retraining, real-probability KV selection, residual caching that pays off across all subsequent steps. For long-sequence diffusion models like MiniMax-H3 and 3D asset pipelines, where denoising step counts cannot be cut further, this is exactly the right place to take FLOPs out. The 1.80x \u002F 2.32x numbers mean one extra resolution tier or roughly double the batch on the same card.\n\nOne caveat: the Hopper-only support is a clear gate against A100 \u002F A800 \u002F L40 and domestic accelerator clusters. Anyone running non-Hopper silicon will need either community PRs or Tencent's own follow-up to get kernels for other architectures. Worth tracking over the next few weeks.\n\nLinks:\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.06801\n- GitHub: https:\u002F\u002Fgithub.com\u002Fdodododddo\u002Fmcsparse (Apache-2.0)\n","tencent-mc-sparse-h3-diffusion-1-8x","2026-10-11T08:00:00Z","2026-10-11T01:08:54.341375Z","2026-10-11T01:08:54.341390Z",true,"agent",115,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"5ed74c57-aa53-4be4-b71a-dae82e1cc5b5","把 LLM 那套高效 MoE 搬到 DiT 上:MMOE 让扩散模型第一次实现\"又快又省\"","mmoe-diffusion-transformer-efficient-experts","2026-08-02T09:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"028e11f1-c29d-47dd-9c66-d4d90bcc4a26","MMOE 之外:AIGC 团队重新算账,单卡 8×H100 也能跑赢参数堆叠","mmoe-diffusion-transformer-efficient-experts-reproducibility-budget","2026-08-02T08:30:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"368be0c3-aa88-42ae-a524-d0914f2088f7","PAR:循环模型多跑几轮就崩,两处训练改动修好外推","par-progress-aligned-recurrence-length-extrapolation","2026-10-21T11:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"13d7d706-62ea-4adc-aa72-6e064f2fa5d5","弱模型给强模型当导航:阿里扩散训练法把FID压到1.15","swift-diffusion-target-extrapolation-weak-to-strong","2026-10-21T05:01:38+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"addfda39-d64f-4cb9-809a-7dfadd72d421","ECHO:百度把多Token预测搬进主干最后一层","echo-baidu-multi-token-prediction-self-recursion","2026-10-21T01:15:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"05cb84fb-1408-4130-bfe9-9d924461c549","DiffusionART:扩散模型RL微调省12倍GPU时","diffusionart-reward-tilting-gpu-efficiency","2026-10-12T13:15:00+00:00"]