On May 11, 2026, a Peking University research team published the MISA paper on arXiv, introducing the MoE routing mechanism into sparse attention, solving the problem where the indexer computation becomes a bottleneck in long-context scenarios with current sparse attention solutions (like DeepSeek DSA).
MISA treats indexer heads as an expert pool, introducing a lightweight router that dynamically selects a small number of active heads to perform token-level scoring based on block-level statistics, rather than having all heads process the full prefix tokens every time. Experimental data shows that with only 8 active heads (DeepSeek-V3.2) and 4 (GLM-5), MISA can match Dense DSA Indexer performance on LongBench, while inference efficiency improves 4-8×. The method requires no additional training and can be deployed directly as a drop-in DSA replacement.
Sparse attention is an important direction for optimizing long-context LLM inference, and MISA's value lies in preserving the diversity of the original indexer pool while significantly reducing compute cost. As LLM context windows continue to grow, this kind of technical approach — fusing MoE routing thinking into the attention mechanism — is becoming a new trend in improving inference efficiency.