Diffusion transformers driving video and high-resolution 3D asset generation are notoriously expensive, and sparse attention is the standard lever for cutting cost — but existing approaches either degrade quality at high sparsity ratios or demand retraining. A 5 October arXiv paper from Tencent Hunyuan (arXiv 2610.06801) proposes a third path: MC-Sparse, a training-free meta-cached sparse attention framework that hits 1.80x denoising speedup on MiniMax-H3-Base and 2.32x on 3D asset generation, with no visible quality loss.

Three root causes the paper pins down

The authors run controlled oracle comparisons and trace existing methods' high-sparsity degradation to three sources:

  • Token grouping constraints — Most sparse-attention implementations force queries or KVs into fixed-size GPU tiles. This adds a granularity the model cannot express in terms of which tokens actually matter;
  • Inaccurate interaction selection — Picking KV tokens via approximate scores or sliding windows consistently misses the ones that contribute real attention mass, with no way to recover later;
  • Lost contributions from discarded tokens — Once a KV is dropped, its share of the dense-attention output is gone for every subsequent denoising step.

Any of these three being mishandled shows up as flickering, blurred details, or facial artifacts in the generated frames.

What MC-Sparse actually does

The framework, Meta-Cached Sparse Attention, attacks each cause:

  • Tile-aligned query groups — Instead of chopping queries into fixed tiles, MC-Sparse uses a Fast PDDP algorithm to cluster similar queries into groups that share GPU scheduling overhead;
  • Exact attention mass for KV selection — At anchor steps the framework actually runs dense attention, then uses the real probability distribution to pick the KV tokens each query truly needs;
  • Cached residuals between dense and sparse outputs — The difference between the dense baseline and the sparse output is computed once, cached as metadata, and reused across all subsequent denoising steps where fresh Q/K/V are computed normally.

The headline is not compute less attention but compute less attention while squeezing every last bit of value out of what you did compute.

Numbers, deployment, and the Hopper ceiling

The reported numbers come straight from the paper and the project page:

  • Video generation on MiniMax-H3-Base: 1.80x denoising speedup, negligible quality loss vs dense attention;
  • 3D asset generation: 2.32x speedup, visibly better geometry preservation than existing sparse baselines.

Code lives at dodododddo/mcsparse under Apache-2.0, with the kernel implementation and a MiniMax-H3 integration included. The README is explicit about one major constraint: only NVIDIA Hopper GPUs are supported. Setup needs a CUDA-enabled PyTorch plus a FlashAttention-3 build exposing flash_attn_interface. The repo ships with several layout scripts:

  • Single-GPU 5s / 15s video generation (run_1gpu_5s.sh / run_1gpu_15s.sh);
  • 7+1 serving-oriented layout (one GPU keeps the text encoder resident while the rest denoise with Ulysses context parallelism);
  • 8-GPU shared layout (encoder returns to CPU after encoding, freeing GPU memory for denoise).

Switches worth knowing: INDEX, OUT_DIR, GPU or NPROC, TOPK or TOPKS, CALIBRATE.

An honest TODO and what is actually shipped

The README TODO list is refreshingly concrete: further optimize Fast PDDP query-grouping speed; expose and validate whitening support in the H3 integration and CLI. What is shipped covers the full MiniMax-H3 sampling pipeline — mcsparse info, mcsparse examples, and mcsparse run drive both video and 3D asset generation end to end. Benchmarks (tests/bench_token_sparse.py, tests/bench_pca_group.py) and a logic-only test suite (tests/test_token_sparse_logic.py) are included, the latter running without PyTorch or CUDA.

Why this matters for diffusion deployment

Sparse attention for video diffusion has been studied for years, but production-grade options have been thin — most papers either change one line of attention and demand a retrain, or push sparsity up until quality collapses. MC-Sparse threads a third route: no retraining, real-probability KV selection, residual caching that pays off across all subsequent steps. For long-sequence diffusion models like MiniMax-H3 and 3D asset pipelines, where denoising step counts cannot be cut further, this is exactly the right place to take FLOPs out. The 1.80x / 2.32x numbers mean one extra resolution tier or roughly double the batch on the same card.

One caveat: the Hopper-only support is a clear gate against A100 / A800 / L40 and domestic accelerator clusters. Anyone running non-Hopper silicon will need either community PRs or Tencent's own follow-up to get kernels for other architectures. Worth tracking over the next few weeks.

Links: