NVIDIA recently released Nemotron 3 Super (120B total parameters / 12B activated), the second-generation open-source reasoning-optimized model following December's Nemotron 3 Nano. Unlike Nano's lightweight positioning, Super directly targets the multi-agent scenario — these systems generate 15× the token volume of normal dialogue per task, with context explosion and "thinking tax" as core pain points. NVIDIA's solution is a complete architectural innovation from attention to deployment.
Latent MoE packs 4× more experts into the same token. Standard MoE routes tokens directly to expert layers; as the model grows, the routing layer itself becomes the compute bottleneck. Nemotron 3 Super introduces Latent MoE: tokens are projected into a low-rank compressed space before entering experts, expert computation happens in a smaller dimension, and the result is projected back to the full dimension. Under the same inference cost, the number of callable experts increases 4×. Finer-grained specialization becomes possible — one expert handles Python syntax, another SQL logic, each activated only when needed.
Hybrid Mamba-Transformer makes linear complexity and precise recall coexist. Nemotron 3 Super's backbone alternates three types of layers: Mamba-2 layers handle the main sequence, introducing the State Space Model's (SSM) linear-time complexity, making 1M-token context practically deployable; Transformer attention layers are inserted at critical depths to ensure high-precision recall on "needle-in-a-haystack" tasks. This combination resolves a long-standing contradiction: SSMs like Mamba are compute-friendly on long sequences but weak on precise-recall; Transformer attention is precise but O(n²) expensive. The hybrid architecture lets each capability play to its strength.
Multi-Token Prediction: two birds, one stone. MTP has the model predict multiple future tokens at each position simultaneously; during training, the model is forced to capture longer-range dependencies, and the multi-token predictions naturally support speculative decoding, reducing autoregressive decoding latency.
Other parameters: native NVFP4 delivers 4× inference speedup on Blackwell; over 1.2 million environment rollouts for multi-environment RL post-training; 85.6% on PinchBench; weights, datasets, and training recipe all open-sourced.
What's really worth attention in Nemotron 3 Super isn't any single technical innovation, but the "efficiency portfolio" design philosophy. Latent MoE solves the routing bottleneck, the Hybrid backbone resolves the context-length vs recall-quality contradiction, and MTP improves both training and inference efficiency — these three layers of innovation interlock, all serving the same goal: making large-parameter models runnable, fast, and accurate on long-horizon multi-agent tasks. NVIDIA's open-source strategy is also reshaping the ecosystem: Nemotron 3 Nano became the standard base for AI Agent deployment shortly after open-sourcing. With Super's bigger efficiency boost and complete training recipe, this path is expected to repeat.