While the industry is still discussing the two routes of quantization and MoE, Sakana AI and NVIDIA have blazed a third trail — unstructured sparsity. The team's latest paper proves that by introducing sparsity in the feed-forward layers (FFN), LLM throughput, energy consumption, and memory footprint can be compressed to a fraction of the original with almost no performance loss.
The LLM's parameters are concentrated in the feed-forward network, accounting for over 70% of parameters and FLOPs. The team, through simple L1 regularization, induced over 99% sparsity in multiple mainstream models — meaning over 99% of FFN parameters can be skipped for most token inferences.
However, unstructured sparsity is hard to execute efficiently on modern GPUs' dense compute pipelines. To address this, the team designed a new sparse-packing format and accompanying CUDA kernels that seamlessly integrate with modern GPU optimized execution pipelines, keeping sparse compute efficient in both training and inference stages.
The paper's most important conclusion: the gains from sparsity increase with model scale. On 70B+ level large models, the number of tokens processable per unit compute increases substantially, and memory-bandwidth pressure eases significantly. This is similar to MoE's characteristic — larger models benefit more from sparsity.
The work was updated to v2 on May 8, 2026, with code open-sourced on GitHub. Against the backdrop of continuously rising LLM inference costs, sparsification is expected to become an important option for the next generation of deployment optimization.