[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-gguf-quant-layout-maps":3,"topics-all":38,"news-related-63042acc-9bec-4d6e-960b-8352ed541bb5":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"63042acc-9bec-4d6e-960b-8352ed541bb5","bartowski 用 1000 组实验重写 GGUF 量化排布规则","bartowski 基于 96 小时 1000+ 组对照实验重写 llama.cpp 量化分配:embedding 敏感度主导、深度呈 U 形、小注意力投影按 bit 最敏感,Q3_K_M 缩小 15%,每个模型形状过金丝雀测试,超标自动回退旧启发式。","给本地大模型下 GGUF 量化文件的人,对 bartowski 这个名字不会陌生:他长期维护一份自己的 llama-quant.cpp 分支,专门改进 MoE 模型的量化处理。9 月 10 日,他在 Hugging Face 博客发布长文,宣布把沿用多年的量化启发式规则整个推翻重写——依据不再是经验,而是约 96 小时、超过 1000 组的对照实验。\n\n## 老规则的天花板\n\nllama.cpp 上游的量化分配代码多年未大改,靠的是模型无关的启发式:比如 `use_more_bits` 检查当前层是否落在前 1\u002F8、后 1\u002F8 或中间每第三层,再决定给哪些张量加码。这套逻辑的依据是几年前的少量测试;MoE 模型专家数超过 8 个就没有特殊处理(上游只写了 Mixtral 恰好 8 专家的特例),小而敏感的 `shexp` 张量也长期被低估。\n\n## 1000 组实验怎么跑\n\n核心方法是\"degrade-one\":把全部张量设为 q8_0,只把其中一个压到 q2_k,逐个测量它单独劣化的代价。指标是量化模型与 bf16 原模型 token 概率之间的 KL 散度(KLD,越低越好),用 `llama-perplexity --kl-divergence` 在 wikitext-2 上以 512 上下文测得。主实验在 Qwen3.5-0.8B 和 4B 上完成,再用 Gemma 4、Granite 4.2、Ling、Muse 等家族交叉验证趋势。实验跑在一台 Framework Desktop 上,测试框架由 Claude 协助编写。\n\n## 三个硬结论\n\n敏感度规律相当清晰:其一,embedding 张量(token_embd)主导整个敏感度尺度,压坏的代价约是最差权重张量的 8 倍(0.8B)到 16 倍(4B),但它从 q4_k 就能找回大部分性能;其二,深度呈 U 形,模型首尾层最怕压缩;其三,小的注意力投影按 bit 算最敏感(attn_v、attn_output、ffn_up、ssm_out 都很突出),而 ffn_gate 基本不值得额外加码。\n\n## 命名重新变得可信\n\n基于这批数据,作者发布了一个 solver:输入模型形状、prior.json 和目标量化类型,输出\"最优\"张量排布。配套的新规则让命名重新有含义——`Q3_K_S` 里 90% 的张量真的是 Q3_K,`_M` 档是 70%,`_L` 档最多 50% 加码。直接效果:Qwen3.5-4B 的 Q3_K_M 缩小 15%;Q4_K_M 在 KLD 略差的情况下小了 5% 以上,按 bit 计算反而更划算。每个新模型形状还要先过\"金丝雀测试\":新方法与旧启发式各出三档量化,画在 KLD-per-bit 曲线上,超标就自动回退旧方法——MiniCPM5-2B 就这样被拦下过。\n\n## 所以呢\n\n对本地推理玩家,影响很直接:同一块显卡,按同样的文件大小去挑新量化,单位 bit 的质量更高;下载 bartowski 新发布的 GGUF 时,模型页新增的 per-tensor layouts 数据值得看一眼。更大的样本意义在于,量化排布这件长期靠手感的事,被拆成了可测量、可复现、可自动校验的工程问题。方法论原文见 [Hugging Face 博客](https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fbartowski\u002Fper-tensor-layout-maps-for-gguf-quantization)。","https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fbartowski\u002Fper-tensor-layout-maps-for-gguf-quantization","341f2956-bab4-471d-8d26-454c68615dd6",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"c715c8e6-3092-4010-887e-e4675a844ce1","en","bartowski rewrites GGUF quant layouts with 1,000 experiments","bartowski rewrote llama.cpp quantization with 1,000+ experiments: embeddings dominate sensitivity, Q3_K_M shrinks 15%, canary tests auto-fallback.","Anyone who downloads GGUF quantized models for local inference knows the name bartowski: for years he has maintained his own fork of llama-quant.cpp, focused on better handling for MoE models. On September 10 he published a long post on the Hugging Face blog, announcing that the old quantization heuristics are being rewritten from scratch — based not on intuition but on roughly 96 hours and more than 1,000 controlled experiments.\n\n## The ceiling of the old rules\n\nThe upstream llama.cpp assignment code has gone largely untouched for years. It relies on model-agnostic heuristics: for example, `use_more_bits` checks whether the current layer falls in the first 1\u002F8th, the last 1\u002F8th, or every third layer in the middle, then decides which tensors get extra bits. That logic traces back to a handful of tests from several years ago. MoE models with more than 8 experts get no special handling at all (upstream only hard-codes the exact Mixtral 8-expert case), and the small but sensitive `shexp` tensors were long underestimated.\n\n## How the 1,000-experiment sweep ran\n\nThe core method is \"degrade-one\": set every tensor to q8_0 except one, which is crushed to q2_k, and measure the damage from that single tensor. The metric is the KL divergence (KLD, lower is better) between the quant's token probabilities and the bf16 model's, measured with `llama-perplexity --kl-divergence` on wikitext-2 at 512 context. The main sweep ran on Qwen3.5-0.8B and 4B, with cross-validation against Gemma 4, Granite 4.2, Ling, Muse, and other families. The machine was a Framework Desktop (AMD AI Max+ 395, 128GB), and the testing framework and scripts were written with Claude's help.\n\n## Three hard findings\n\nThe sensitivity patterns are remarkably clear. First, the embedding tensor (token_embd) dominates the entire sensitivity scale — crushing it costs about 8x the worst single weight tensor at 0.8B and 16x at 4B, though it recovers most of its performance from q4_k. Second, depth forms a U shape: the first and last layers of the model are the most compression-averse. Third, small attention projections are the most sensitive per bit (attn_v, attn_output, ffn_up, and ssm_out all stand out), while ffn_gate is basically never worth extra bits.\n\n## Names become trustworthy again\n\nFrom this data the author shipped a solver: feed it a model shape, the prior.json table, and a target quant type, and it outputs the \"optimal\" tensor layout. The accompanying naming rules restore meaning to the file names — in a `Q3_K_S`, 90% of the tensors are genuinely Q3_K; `_M` means 70%; `_L` allows up to 50% bumped. Direct effects: Q3_K_M on Qwen3.5-4B shrinks by 15%; Q4_K_M comes in just over 5% smaller with a marginally worse KLD, meaning better value per bit. Every new model shape must first pass a \"canary test\": both the new method and the old heuristic produce Q4_K_M, Q3_K_M, and IQ2_XS, plotted on a KLD-per-bit curve — if the mapped quants fall above the curve beyond noise, the pipeline automatically falls back to the old heuristic. MiniCPM5-2B was caught this way.\n\n## So what\n\nFor local inference users the impact is immediate: on the same GPU, picking new quants by the same file size gets you higher quality per bit; when downloading bartowski's freshly released GGUFs, the new per-tensor layouts data on the model page is worth a look. The bigger takeaway: quantization layout, long a matter of feel, has been decomposed into a measurable, reproducible, and automatically verifiable engineering problem. Full methodology on the [Hugging Face blog](https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fbartowski\u002Fper-tensor-layout-maps-for-gguf-quantization).","gguf-quant-layout-maps","2026-09-10T21:05:00Z","2026-09-10T21:10:14.304188Z","2026-09-10T21:10:14.304203Z",true,"agent",100,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"34b087d5-f502-4a46-9599-6f014fe0cf89","llama.cpp v0.4.0 发布:权重懒加载进主线,小内存也能碰 75B 级 MoE","llama-cpp-v0-4-0-lazy-tensor-loading","2026-09-07T13:10:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"988bbfb8-672a-4c6c-98f7-3a170b6bd8b3","Macaw 把 LFM2.5 装进 1.5GB:4-bit 端侧 LLM 跑 Mac 控制工具链","macaw-lfm25-15gb-edge-mac-agent","2026-08-24T06:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"d5203c93-2022-4769-a6d2-c7765ded2b40","腾讯混元 Hunyuan-A13B 开源实测:80B 总参 \u002F 13B 激活,GQA + FP8\u002FINT4 把 MoE 推理门槛打到消费卡","tencent-hunyuan-a13b-80b-13b-gqa-angelslim-moe","2026-07-30T06:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1abe59b8-844d-4bb1-bc27-cfe28099d101","Anemll\u002FFlash-iOS：把 400B MoE 大模型塞进 iPhone 的端侧实验","anemll-flash-ios-400b-moe-iphone-edge","2026-06-07T12:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"c2e18ecb-00b3-47ee-935a-f0e3cd0dee4a","Holo3.1 把 Computer Use Agent 拉进本地：FP8\u002FNVFP4\u002FQ4 三种量化让消费级 GPU 跑得动","holo3-1-h-company-computer-use-quant","2026-06-07T06:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"8c5bd0e1-551a-405b-b2fc-a67520bf84c5","Colibri v1.11.0 发布:纯 C 引擎直读 DeepSeek V4.1 Flash,552B 从 SSD 流进 CPU","colibri-v1-11-deepseek-v41-flash","2026-09-14T13:09:40+00:00"]