Background: The Ceiling of Hand-Tuned Inference

For local LLM inference, llama.cpp has been the de facto standard for years. The default workflow is well known: clone ggml-org, grab a GGUF, run llama-bench. But llama.cpp's speed comes almost entirely from hand-tuned kernels — every new fusion or architecture means writing another kernel by hand. When model iteration outpaces hand-tuning, that is the ceiling.

In October 2026, the HardenedLinux community released SonicBoom v0.0.3 — an AI runtime that puts an MLIR compiler alongside traditional quantized kernels. The core is only 22k lines of C++/CUDA, exposed through a stable C ABI, GPLv3-licensed. The GitHub repo hardenedlinux/SonicBoom leads with one line: "A native tensor runtime. No Python. One .so."

The thesis is clear: stop hand-writing kernels, let the compiler generate them.

The Architecture: A Compiler Stack, Not a Kernel Collection

SonicBoom is layered: Layer 0 is CPU/CUDA hardware, Layer 1 is the native-torch implementation, Layer 2 is the stable runtime interface, Layer 3 is the C ABI plus language bindings. Dependencies flow strictly downward. libsonicboom.so is the published artifact, with MLIR statically linked in.

The runtime has three tiers:

  • MLIR compiler tier. Model graphs are lowered into Boom IR, rewritten by MLIR passes, then taken through linalg → gpu → nvvm to produce GPU kernels. These are JIT-compiled and cached by shape signature.
  • Graph planner tier. Before any model runs, the whole graph is rewritten: quantize-once, norm→quantize fusion, activation→quantize fusion, dead-computation elimination, layout selection, and CUDA Graph capture of the full decode step.
  • Hand-written kernels. Used only where the other tiers have not caught up — for example, the MMQ/MMVQ kernels ported from ggml and the tiled prefill attention.

In other words, SonicBoom is no longer a "kernel collection" — it is a compiler stack.

Key Numbers: Same Kernels, Faster Runtime

v0.0.3 verifies two model families: Gemma 4 E4B-it Q3_K_M and Qwen2.5-0.5B-Instruct Q4_K_M, both on RTX 3050 8GB against llama.cpp at commit 43fe9c64281.

Gemma 4 E4B-it Q3_K_M:

  • Decode 64-step steady state: 53.0 tok/s vs 44.8 tok/s, 1.18×
  • Decode at KV depth 512: 52.1 tok/s vs 44.2 tok/s, 1.18×
  • Prefill 512-token prompt: 1990 tok/s vs 1992 tok/s, 1.00× (parity)

Qwen2.5-0.5B-Instruct Q4_K_M:

  • Decode 64-step steady state: 344.7 tok/s vs 307.6 tok/s, 1.12×
  • Prefill 512-token prompt: 15346 tok/s vs 14044 tok/s, 1.09×

The crucial fact: decode uses llama.cpp's quantized GEMV kernels, untouched. The 12–18% lead comes entirely from the runtime — the planner and the MLIR tier. This is the proof of the README's claim: "fast because of the runtime, not only the kernels."

Prefix-cache reuse is also worth noting: on the same session, if a later prompt starts with the same tokens, only the new suffix is prefilled. The Gemma latency is 65.2 ms to first token for a 512-token prefix plus 64 new tokens; Qwen2 is 13.7 ms. llama-bench cannot measure this, so the README reports it directly as first-token latency.

Compiler Details: From S-Expr to NVVM

GNU Guile is SonicBoom's first-class language. Delimited continuations drive "dead computation elimination" at the application layer — only effective computation enters the runtime. Researchers can write models in Scheme, trace them into a graph, and feed them to the planner.

The native-torch operator library is generated at build time by SonicCross, a Guile-written torchgen replacement. It converts PyTorch's native_functions.yaml into native-torch's C++ declarations, registrations, and dispatch sources — over 7,000 per-operator headers per run. SonicCross only runs at build time and is not linked into libsonicboom.so.

LLVM/MLIR is pinned to major version 23 and statically linked. Users with an existing LLVM install can point at it via -DMLIR_DIR=/opt/llvm/lib/cmake/mlir and skip the submodule sync.

Contrast With PyTorch: No Python Required

The README states the position directly: SonicBoom is "a better PyTorch, with no Python in it."

  • The same eager style, without the Python interpreter, pip, or wheel tree.
  • The same model authoring, but the production path runs through a C++ core while the research path runs through Scheme.
  • The same flexibility, but instead of a CPython bytecode interpreter, SonicBoom uses Guile's syntax-rules macros and delimited continuations. TorchDynamo needs the CPython interpreter to approach "data-dependent control flow without graph breaks"; SonicBoom does it with macros.

For deployment, this matters. The README describes a single ~300 MB .so file — none of the dozens-of-gigabytes Python stack that llama.cpp users accept today.

Status and Roadmap

v0.0.3 is not a drop-in replacement for llama.cpp. The boundaries are explicit:

  • Two verified model families: Gemma 4 E4B-it Q3_K_M and Qwen2.5-0.5B-Instruct Q4_K_M. Other models can run, but community reproductions are required.
  • Quant formats: Q3_K, Q4_K, Q5_K, Q6_K, Q5_0, Q8_0; f16 and q4_0 are converted to Q5_K at load.
  • Sampling: greedy only; temperature, top_p, top_k all return SB_ERR_UNSUPPORTED.
  • GPU: CUDA sm_86 (RTX 30-series); one community report has reproduced decode numbers on RTX 4060 Ti Ada.

Notable roadmap items:

  • Prefill is still attention-bound; the next lever is a generated tensor-core FA-2.
  • The Vulkan backend is already in the tree (-DSONICBOOM_USE_VULKAN=ON), targeting non-NVIDIA and fully free-software stacks.
  • The roadmap also lists "Beyond LLMs" — diffusion models, with a ComfyUI workflow converter planned.

Why This Matters

The key bet is not "faster than llama.cpp" but "kernels should not be the limit." llama.cpp's design philosophy is hand-tuned kernels; every new fusion means another kernel. SonicBoom's design philosophy is "write the pass once, every architecture benefits" — planner and compiler carry the cross-model optimization load.

This mirrors the path the training stack already took, moving from hand-written CUDA kernels to compiler stacks (Triton, TVM, the MLIR family). The inference stack is finally catching up.

The repo sits at 10 stars and 271 commits today, but the company side — HardenedVault, the corporate vehicle of the HardenedLinux community — offers commercial consulting on private/air-gapped LLM inference, performance tuning, and hardened inference stacks.

If your hardware is RTX 30-series or newer NVIDIA and you have a GGUF model, it is worth spending a night getting it running. The repo is at hardenedlinux/SonicBoom.

References: HardenedLinux SonicBoom repo / Solidot writeup