A graphics card with 24GB of VRAM used to struggle with 30B-class models. NVIDIA just open-sourced a 74B-parameter MoE that fits inside that budget, with a hard floor of 80 tokens per second on decode speed. Nemotron 3.5 Bolt 74B-A9B 24GB is a hybrid Mamba-Transformer built for everyday agentic tasks on RTX and DGX Spark — machines where the model has to coexist with everything else you run.
Compressed from 120B, not trained from scratch
Bolt is not trained from zero: it is compressed from its bigger sibling Nemotron 3.5 Super VL (120B total, 12B active). Total parameters drop from 120B to 74B, active parameters from 12B to 9B, and the expert top-K per token falls from 22 to 9. The tool is Star Elastic, a push-button, router-guided architecture-search framework: a budget-conditioned router picks how many routed experts each layer keeps and how wide the MoE intermediates should be, targeting a sub-24GB deployment footprint with downstream quantization priced in. NVIDIA's experiments land on a counterintuitive conclusion — for a fixed memory budget, compressing a large model beats training a smaller one from scratch and quantizing it to 4 bits. After compression, additional post-training with end-to-end reinforcement learning in simulated environments recovers vision functionality and local-agent capability.
The 2-bit bet
The aggressive part is quantization. The FC1/FC2 expert layers — the bulk of the weights — are quantized to IQ2_XXS at 2.0625 bits per element. Mamba linear layers use NVFP4, while the lm_head, token embeddings, attention, and latent-MoE weights stay at Q8_0. The rationale is explicit: inference on RTX/DGX Spark is mostly weight-bandwidth dominated, so weight bit-width directly sets how many parameters per second the chip can feed. The accuracy lost to quantization is bought back with quantization-aware distillation (QAD). Behind this sits the ParetoQ and Quest line of research, which argued that at equal memory, large networks at 1-2 bits are more Pareto-optimal than smaller networks at 4 bits. Bolt turns that academic claim into a downloadable checkpoint.
Speed and leaderboards
The bundled LiLiCorr speculative-decoding drafter has only 1.13B parameters and jointly scores token candidates across an entire draft in one lightweight pass; in vLLM with seven drafted tokens it reaches a mean acceptance length of 4.68 under greedy decoding. On RTX Spark, the full stack hits 84 TPS on the AIPerf SPEED-Bench. In NVIDIA's own comparison, the two closest 24GB-class rivals — Qwen3.6 35B-A3B and Gemma 4 26B-A4B — trail on benchmarks for general agentic capability, web browsing, and terminal use. On public leaderboards, NVIDIA reports Bolt at #1 on PinchBench and #2 on both WebArena and BrowseComp among models with a similar memory footprint. Treat these as vendor-reported numbers until independently reproduced.
A 4GB sibling lands on October 23
Per IT Home, NVIDIA also has Bolt E4B 4GB on the way, aimed at local voice assistants, interactive characters, and embedded interfaces, planned for October 23. From 24GB down to 4GB, NVIDIA is turning local-agent memory tiers into a product line, supported across llama.cpp, vLLM, Ollama, Unsloth, and LM Studio, with both the quantized and BF16 checkpoints released on Hugging Face.
So what
The most memorable thing about Bolt is not a leaderboard rank but the strategic question it forces: for local agents, do you want small-and-dense, or huge-and-sparse with ultra-low bit-widths? NVIDIA bet on the latter by stuffing 74B into 24GB. If 2-bit weights hold up on everyday agent workloads, then under a fixed memory budget the binding constraint stops being how many parameters fit — it becomes how fast the bandwidth can feed them. Weight bit-width is turning into the new currency of on-device models.
References: NVIDIA technical page (https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/); IT Home coverage (https://www.ithome.com/1/013/679.htm)