[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sonicboom-v003-mlir-compiler":3,"topics-all":44,"news-related-ab1e6bdf-9433-41f0-be4d-cc0f78523ac2":63},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":30,"news_slug":37,"published_at":38,"created_at":39,"modified_at":40,"is_published":41,"publish_type":42,"image_url":14,"view_count":43},"ab1e6bdf-9433-41f0-be4d-cc0f78523ac2","SonicBoom v0.0.3:一个 .so 干掉手工调优","SonicBoom v0.0.3 由 HardenedLinux 社区发布,GPLv3。它以 LLVM\u002FMLIR 为唯一重写路径,在 RTX 3050 上复用 llama.cpp 同款量化 kernel,Gemma 4 与 Qwen2.5 decode 比 llama.cpp 快 1.18× 与 1.12×。","## 技术背景:本地 LLM 推理的\"内卷\"拐点\n\n本地跑 LLM 这几年,llama.cpp 几乎成了\"标准答案\"。开发者社区里,默认动作就是去 GitHub 克隆 ggml-org 仓库,拿 GGUF 权重,塞进 llama-bench 看 decode 速度。但 llama.cpp 的速度来源,基本是工程师手写 kernel 的极限优化——每加一个新融合、一个新模型架构,就要写一个新 kernel。当模型迭代速度赶上不上手写速度,这就是天花板。\n\n2026 年 10 月,HardenedLinux 社区推出了 SonicBoom v0.0.3——一个把 MLIR 编译器与传统量化 kernel 放在一起的 AI 运行时框架。它只有 22k 行 C++\u002FCUDA 核心,稳定 C ABI 暴露能力,GPLv3 协议开源。GitHub 仓库 hardenedlinux\u002FSonicBoom 在 README 里就一句话:\"A native tensor runtime. No Python. One .so.\"\n\n它的思路非常清晰:不要再去手写 kernel 了,直接让编译器生成。\n\n## 核心内容:把编译器搬进运行时\n\nSonicBoom 的架构是四层分层,Layer 0 是 CPU\u002FCUDA 硬件,Layer 1 是 native-torch 实现,Layer 2 是稳定运行时接口,Layer 3 是 C ABI 与多语言绑定(Layer 依赖严格向下)。整个 `libsonicboom.so` 是对外发布的库,MLIR 是静态链接进去的,不是运行时依赖。\n\n它的核心不是单点创新,而是组合拳:\n\n- **MLIR 编译器层**:模型图先被 lowering 到 Boom IR,然后经过 MLIR 的 rewrite passes,最终通过 `linalg → gpu → nvvm` 生成 GPU kernel。这些 kernel 是 JIT 编译的,按 shape signature 缓存。\n- **图规划器层**:在模型真正运行之前,整个模型会被重写一遍——量化-一次到位、norm→quantize 融合、激活→量化融合、死计算消除、layout 选择、整段 decode 步骤的 CUDA Graph capture。\n- **手写 kernel 层**:只在编译器还没追上的地方兜底——比如从 ggml 移植过来的 MMQ\u002FMMVQ 量化矩阵乘和 tiled prefill attention。\n\n也就是说,SonicBoom 不再是一个\"kernel 集合\",而是一个\"编译器栈\"。\n\n## 关键性能:同一 kernel,不同框架\n\nv0.0.3 验证了两个模型家族:Gemma 4 E4B-it Q3_K_M 和 Qwen2.5-0.5B-Instruct Q4_K_M,均在 RTX 3050 8GB 上对照 llama.cpp commit 43fe9c64281 跑出来的数据。\n\n**Gemma 4 E4B-it Q3_K_M**:\n- Decode 64-step steady state:53.0 tok\u002Fs vs 44.8 tok\u002Fs,**1.18×**\n- Decode at KV depth 512:52.1 tok\u002Fs vs 44.2 tok\u002Fs,**1.18×**\n- Prefill 512-token prompt:1990 tok\u002Fs vs 1992 tok\u002Fs,**1.00×**(持平)\n\n**Qwen2.5-0.5B-Instruct Q4_K_M**:\n- Decode 64-step steady state:344.7 tok\u002Fs vs 307.6 tok\u002Fs,**1.12×**\n- Prefill 512-token prompt:15346 tok\u002Fs vs 14044 tok\u002Fs,**1.09×**\n\n关键的事实是:decode 用的就是 llama.cpp 的量化 GEMV kernel,优化没有改。那 12–18% 的领先完全来自 runtime 层——图规划器和 MLIR 编译器层的优化。这印证了 README 里那句\"says: fast because of the runtime, not only the kernels\"。\n\n另一个工程亮点是 prefix-cache reuse:同一个 session 后续 prompt 如果以相同 token 开头,只有新增的后缀会被 prefill。一个 512-token prefix + 64 new 的延迟:Gemma 65.2 ms,Qwen2 13.7 ms。`llama-bench` 量不出这个,所以 README 直接以\"first-token latency in milliseconds\"报。\n\n## 编译器层细节:从 S-Expr 到 NVVM\n\nSonicBoom 用 GNU Guile 作为 first-class 语言,这一选择让应用层可以享受 PyTorch 一样的灵活性——delimited continuation 实现\"死计算消除\",只有真正有效的计算才进入运行时。研究侧的模型可以直接用 Scheme 写,然后 trace 到一张图里,送进 planner。\n\nnative-torch 的 operator 库在编译时由 SonicCross 生成——这是一个用 Guile 写的 torchgen 替代品。它把 PyTorch 的 `native_functions.yaml` 转成 native-torch 的 C++ declaration、registration、dispatch sources,每轮生成超过 7,000 个 per-operator 头文件。SonicCross 只在 build time 运行,不链接进 `libsonicboom.so`。\n\nLLVM\u002FMLIR 锁版本是 23,被静态链接进库。如果用户已有 LLVM 安装,可以通过 `-DMLIR_DIR=\u002Fopt\u002Fllvm\u002Flib\u002Fcmake\u002Fmlir` 等 CMake 变量指向,跳过 submodule 同步。\n\n## 与 PyTorch 的对照:一个不装 Python 的版本\n\nREADME 里直接点破:SonicBoom 想做的是 \"a better PyTorch, with no Python in it\"。\n\n- 同样的 eager 风格,但不需要 Python 解释器、不需要 pip、不需要 wheel tree。\n- 同样的模型定义方式,但生产路径用的是 C++ core,研究路径用的是 Scheme。\n- 同样的灵活性,但底层不再是 Python bytecode 解释器,而是 Guile 的 `syntax-rules` 和 delimited continuation——TorchDynamo 需要 CPython 字节码解释器才能解决\"data-dependent control flow without graph breaks\",SonicBoom 用宏直接做。\n\n这套路线对生产部署非常友好。README 给了一个三百多 MB 的 `.so` 文件,部署成本压到极限,没有几十 G 的 Python 全家桶。\n\n## 现状与路线图\n\nv0.0.3 不是 llama.cpp 的 drop-in 替代。明确边界:\n\n- 两个验证模型家族:Gemma 4 E4B-it Q3_K_M 与 Qwen2.5-0.5B-Instruct Q4_K_M。其他模型能用,但需要社区跑出复现数据。\n- 量化格式:Q3_K、Q4_K、Q5_K、Q6_K、Q5_0、Q8_0,`f16` 和 `q4_0` 在 load 时转 Q5_K。\n- 采样:仅 greedy,`temperature`、`top_p`、`top_k` 都返回 `SB_ERR_UNSUPPORTED`。\n- GPU:CUDA sm_86(RTX 30 系),社区已有 RTX 4060 Ti Ada 上的复现数据。\n\n路线图上比较值得关注的:\n\n- prefill 还有 attention 是瓶颈,下一个 lever 是 generated tensor-core FA-2。\n- Vulkan 后端已经在 tree 里(`-DSONICBOOM_USE_VULKAN=ON`),面向 non-NVIDIA 与纯自由软件用户。\n- 路线图上还有 \"Beyond LLMs\"——diffusion 模型,准备做一个 ComfyUI workflow 转换器。\n\n## 行业意义:不是又一个 llama.cpp\n\nSonicBoom 的关键判断不是\"比 llama.cpp 更快\",而是\"kernel 不该是 limit\"。llama.cpp 的设计哲学是 hand-tuned kernels,每加一个 fusion 就写一个新 kernel。SonicBoom 的设计哲学是\"一次写 pass,所有架构都受益\",由 planner 与编译器承担跨模型的优化。\n\n这跟大模型训练框架从手写 CUDA kernel 切到编译器栈(Triton、TVM、MLIR 派系)的路径高度相似。推理栈这次终于跟上了。\n\n社区目前 10 stars、271 commits,但合作方 HardenedVault(HardenedLinux 社区的公司化主体)提供企业咨询,业务覆盖 private\u002Fair-gapped LLM inference、性能调优、hardened 推理栈——也就是把它当作一个可商业支持的 runtime 来做。\n\n如果你的卡是 RTX 30 系或更新的 NVIDIA,手里有 GGUF 模型,值得花一个晚上把它跑起来。GitHub 仓库就在 hardenedlinux\u002FSonicBoom。\n\n参考链接:[HardenedLinux SonicBoom 仓库](https:\u002F\u002Fgithub.com\u002Fhardenedlinux\u002FSonicBoom) \u002F [Solidot 中文介绍](https:\u002F\u002Fwww.solidot.org\u002Fstory?sid=85645)","https:\u002F\u002Fwww.solidot.org\u002Fstory?sid=85645","d59894d3-308e-4fd8-8865-86dc1eeac4a2",[11,15,18,21,24,27],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"fca9258a-9430-455a-b95d-b9fae5e373a8","ai-inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":25,"name":26,"slug":26,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":28,"name":29,"slug":29,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[31],{"id":32,"lang":33,"title":34,"summary":35,"content":36},"94867316-1a6a-4431-b389-dfa19de8598d","en","SonicBoom v0.0.3: An MLIR Compiler Stack for Local LLM","SonicBoom v0.0.3: 22k lines C++\u002FCUDA, stable C ABI, MLIR compiler reuses llama.cpp kernels. Gemma 4 decodes 1.18x faster on RTX 3050.","## Background: The Ceiling of Hand-Tuned Inference\n\nFor local LLM inference, llama.cpp has been the de facto standard for years. The default workflow is well known: clone ggml-org, grab a GGUF, run llama-bench. But llama.cpp's speed comes almost entirely from hand-tuned kernels — every new fusion or architecture means writing another kernel by hand. When model iteration outpaces hand-tuning, that is the ceiling.\n\nIn October 2026, the HardenedLinux community released SonicBoom v0.0.3 — an AI runtime that puts an MLIR compiler alongside traditional quantized kernels. The core is only 22k lines of C++\u002FCUDA, exposed through a stable C ABI, GPLv3-licensed. The GitHub repo hardenedlinux\u002FSonicBoom leads with one line: \"A native tensor runtime. No Python. One .so.\"\n\nThe thesis is clear: stop hand-writing kernels, let the compiler generate them.\n\n## The Architecture: A Compiler Stack, Not a Kernel Collection\n\nSonicBoom is layered: Layer 0 is CPU\u002FCUDA hardware, Layer 1 is the native-torch implementation, Layer 2 is the stable runtime interface, Layer 3 is the C ABI plus language bindings. Dependencies flow strictly downward. `libsonicboom.so` is the published artifact, with MLIR statically linked in.\n\nThe runtime has three tiers:\n\n- **MLIR compiler tier.** Model graphs are lowered into Boom IR, rewritten by MLIR passes, then taken through `linalg → gpu → nvvm` to produce GPU kernels. These are JIT-compiled and cached by shape signature.\n- **Graph planner tier.** Before any model runs, the whole graph is rewritten: quantize-once, norm→quantize fusion, activation→quantize fusion, dead-computation elimination, layout selection, and CUDA Graph capture of the full decode step.\n- **Hand-written kernels.** Used only where the other tiers have not caught up — for example, the MMQ\u002FMMVQ kernels ported from ggml and the tiled prefill attention.\n\nIn other words, SonicBoom is no longer a \"kernel collection\" — it is a compiler stack.\n\n## Key Numbers: Same Kernels, Faster Runtime\n\nv0.0.3 verifies two model families: Gemma 4 E4B-it Q3_K_M and Qwen2.5-0.5B-Instruct Q4_K_M, both on RTX 3050 8GB against llama.cpp at commit 43fe9c64281.\n\n**Gemma 4 E4B-it Q3_K_M**:\n- Decode 64-step steady state: 53.0 tok\u002Fs vs 44.8 tok\u002Fs, **1.18×**\n- Decode at KV depth 512: 52.1 tok\u002Fs vs 44.2 tok\u002Fs, **1.18×**\n- Prefill 512-token prompt: 1990 tok\u002Fs vs 1992 tok\u002Fs, **1.00×** (parity)\n\n**Qwen2.5-0.5B-Instruct Q4_K_M**:\n- Decode 64-step steady state: 344.7 tok\u002Fs vs 307.6 tok\u002Fs, **1.12×**\n- Prefill 512-token prompt: 15346 tok\u002Fs vs 14044 tok\u002Fs, **1.09×**\n\nThe crucial fact: decode uses llama.cpp's quantized GEMV kernels, untouched. The 12–18% lead comes entirely from the runtime — the planner and the MLIR tier. This is the proof of the README's claim: \"fast because of the runtime, not only the kernels.\"\n\nPrefix-cache reuse is also worth noting: on the same session, if a later prompt starts with the same tokens, only the new suffix is prefilled. The Gemma latency is 65.2 ms to first token for a 512-token prefix plus 64 new tokens; Qwen2 is 13.7 ms. `llama-bench` cannot measure this, so the README reports it directly as first-token latency.\n\n## Compiler Details: From S-Expr to NVVM\n\nGNU Guile is SonicBoom's first-class language. Delimited continuations drive \"dead computation elimination\" at the application layer — only effective computation enters the runtime. Researchers can write models in Scheme, trace them into a graph, and feed them to the planner.\n\nThe native-torch operator library is generated at build time by SonicCross, a Guile-written torchgen replacement. It converts PyTorch's `native_functions.yaml` into native-torch's C++ declarations, registrations, and dispatch sources — over 7,000 per-operator headers per run. SonicCross only runs at build time and is not linked into `libsonicboom.so`.\n\nLLVM\u002FMLIR is pinned to major version 23 and statically linked. Users with an existing LLVM install can point at it via `-DMLIR_DIR=\u002Fopt\u002Fllvm\u002Flib\u002Fcmake\u002Fmlir` and skip the submodule sync.\n\n## Contrast With PyTorch: No Python Required\n\nThe README states the position directly: SonicBoom is \"a better PyTorch, with no Python in it.\"\n\n- The same eager style, without the Python interpreter, pip, or wheel tree.\n- The same model authoring, but the production path runs through a C++ core while the research path runs through Scheme.\n- The same flexibility, but instead of a CPython bytecode interpreter, SonicBoom uses Guile's `syntax-rules` macros and delimited continuations. TorchDynamo needs the CPython interpreter to approach \"data-dependent control flow without graph breaks\"; SonicBoom does it with macros.\n\nFor deployment, this matters. The README describes a single ~300 MB `.so` file — none of the dozens-of-gigabytes Python stack that llama.cpp users accept today.\n\n## Status and Roadmap\n\nv0.0.3 is not a drop-in replacement for llama.cpp. The boundaries are explicit:\n\n- Two verified model families: Gemma 4 E4B-it Q3_K_M and Qwen2.5-0.5B-Instruct Q4_K_M. Other models can run, but community reproductions are required.\n- Quant formats: Q3_K, Q4_K, Q5_K, Q6_K, Q5_0, Q8_0; `f16` and `q4_0` are converted to Q5_K at load.\n- Sampling: greedy only; `temperature`, `top_p`, `top_k` all return `SB_ERR_UNSUPPORTED`.\n- GPU: CUDA sm_86 (RTX 30-series); one community report has reproduced decode numbers on RTX 4060 Ti Ada.\n\nNotable roadmap items:\n\n- Prefill is still attention-bound; the next lever is a generated tensor-core FA-2.\n- The Vulkan backend is already in the tree (`-DSONICBOOM_USE_VULKAN=ON`), targeting non-NVIDIA and fully free-software stacks.\n- The roadmap also lists \"Beyond LLMs\" — diffusion models, with a ComfyUI workflow converter planned.\n\n## Why This Matters\n\nThe key bet is not \"faster than llama.cpp\" but \"kernels should not be the limit.\" llama.cpp's design philosophy is hand-tuned kernels; every new fusion means another kernel. SonicBoom's design philosophy is \"write the pass once, every architecture benefits\" — planner and compiler carry the cross-model optimization load.\n\nThis mirrors the path the training stack already took, moving from hand-written CUDA kernels to compiler stacks (Triton, TVM, the MLIR family). The inference stack is finally catching up.\n\nThe repo sits at 10 stars and 271 commits today, but the company side — HardenedVault, the corporate vehicle of the HardenedLinux community — offers commercial consulting on private\u002Fair-gapped LLM inference, performance tuning, and hardened inference stacks.\n\nIf your hardware is RTX 30-series or newer NVIDIA and you have a GGUF model, it is worth spending a night getting it running. The repo is at hardenedlinux\u002FSonicBoom.\n\nReferences: [HardenedLinux SonicBoom repo](https:\u002F\u002Fgithub.com\u002Fhardenedlinux\u002FSonicBoom) \u002F [Solidot writeup](https:\u002F\u002Fwww.solidot.org\u002Fstory?sid=85645)","sonicboom-v003-mlir-compiler","2026-10-19T09:03:05Z","2026-10-19T09:05:04.318393Z","2026-10-19T09:05:04.318409Z",true,"agent",80,[45,54],{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":41,"created_at":52,"modified_at":53},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":55,"tag_slug":55,"title_zh":56,"title_en":57,"intro_zh":58,"intro_en":59,"id":60,"is_active":41,"created_at":61,"modified_at":62},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":64},[65,70,75,80,85,90],{"id":66,"title":67,"news_slug":68,"published_at":69},"4ccc491b-dbee-4c84-beb6-7cf519f76320","LoRA 基座换 GGUF:40G 显存训 125B","lora-over-gguf-low-vram-training","2026-10-10T21:08:22+00:00",{"id":71,"title":72,"news_slug":73,"published_at":74},"9a77e0d7-8164-494e-984d-54cb57a7a0dd","Magnitude 开源:本地 Agent 专用推理引擎","magnitude-self-tuning-local-agent-inference-engine","2026-10-01T17:08:35+00:00",{"id":76,"title":77,"news_slug":78,"published_at":79},"b571067a-9fa8-42bf-9431-98f26ac78e03","伯克利把LLM推理搬进SSD:KV缓存压缩15倍","llm-inference-in-flash-cim-ssd","2026-09-19T21:10:00+00:00",{"id":81,"title":82,"news_slug":83,"published_at":84},"63042acc-9bec-4d6e-960b-8352ed541bb5","bartowski 用 1000 组实验重写 GGUF 量化排布规则","gguf-quant-layout-maps","2026-09-10T21:05:00+00:00",{"id":86,"title":87,"news_slug":88,"published_at":89},"34b087d5-f502-4a46-9599-6f014fe0cf89","llama.cpp v0.4.0 发布:权重懒加载进主线,小内存也能碰 75B 级 MoE","llama-cpp-v0-4-0-lazy-tensor-loading","2026-09-07T13:10:00+00:00",{"id":91,"title":92,"news_slug":93,"published_at":94},"42b7939c-1b44-43b8-95cf-a8fc2204560d","NVIDIA 开源 Personal AI Router，把家里 RTX 与 Mac 拼成本地 AI 集群","nvidia-personal-ai-router-pair-beta","2026-09-04T03:20:00+00:00"]