[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-quantization-bits-vs-option-order-budget":3,"topics-all":38,"news-related-081148f2-39eb-432d-8841-5cb1e58ed527":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"081148f2-39eb-432d-8841-5cb1e58ed527","省内存还是多跑几遍:本地小模型怎么花推理预算才划算","本地跑小模型,量化省内存和多跑几遍取平均会互相干扰。英特尔团队在 Qwen3-0.6B\u002F1.7B 上把两者折算成同一单位(权重字节×遍数):6 比特一遍只读 0.391 的 fp16 权重却不掉点,4 比特却让模型按答案位置乱答,78% 落在 A 槽。决策表收敛成一句:6 比特打底,余下预算换最多四个顺序平均。","把本地小模型量化到 4 位省内存,还是让它多跑几遍取平均提分?这像常识题,但两者会互相干扰——量化会改变模型对选项顺序的敏感度,而多跑几遍取平均恰恰是治这个的常规药。11 月 10 日提交到 arXiv 的一篇论文([arXiv:2611.14413](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14413))把两者放进同一本账里算,作者来自 Intel。\n\n## 实验设定:两种省钱手段,四张多选基准\n\n模型是 Qwen3-0.6B 和 Qwen3-1.7B,基准取 ARC-Easy、ARC-Challenge、MMLU 和 HellaSwag,每任务 300 题,每模型合计 1200 题。量化用最朴素的 round-to-nearest(absmax 缩放,每 64 个权重一组 fp16 标度),位宽取 8、6、4、3 比特,fp16 作参照;权重只在内存里模拟舍入,不打包也不跑低比特 kernel,embedding 和 norm 保持 fp16。每题跑 8 种选项顺序:原顺序的 4 个循环移位加反序的 4 个。\n\n成本折算成一个与硬件无关的单位:权重字节数乘前向遍数。8 位一遍 0.516,6 位 0.391,4 位 0.266,3 位 0.203,fp16 是 1。\n\n## 位宽账:6 比特是甜点,4 比特开始塌\n\n6 比特相对 fp16 不掉点,反而高 1.8 分(0.6B)和 0.9 分(1.7B),但作者明说没找到原因,也不宣称 6 比特能提升模型,只按「不亏」来读。8 比特分别掉 0.7 分和 0.4 分。4 比特开始塌:0.6B 掉 14.3 分,1.7B 掉 5.6 分。3 比特两个模型都趴在随机水平——25.4% 和 26.6%,而四选一随机基线是 25%。\n\n## 4 比特不只是变笨,它开始按位置答\n\n这是全文最有意思的发现。论文用 slot skew 衡量「答案落在哪个字母槽」偏离均匀分布的程度:0.6B 在 fp16 下是 0.15,1.7B 是 0.07;到 4 比特,0.6B 飙到 0.53,1.7B 是 0.12。具体表现是 4 比特的 0.6B 有 78% 的答案落在 A 槽——它不再是在答题,是在按位置答题,同槽位重复率也从 0.24 升到 0.70。\n\n后果是:正确选项恰好落在它偏爱的槽位时,4 比特模型答得远好于平均,别处则远差于平均,四个 gold 槽的准确率跨度从 fp16 的 0.28 拉大到 0.76。基准跑分把这个偏置稀释掉了,但你自己的问题若答案位置有偏,分数会是另一个数。\n\n作者也做了对照:同等信噪比的高斯噪声扰动权重,0.6B 两次抽样给出 0.15 和 0.37,1.7B 一次给出 0.19。控制实验不确定,没法断定这种位置偏置是「取整」独有的——噪声也能造出来。\n\n## 换算表:比特怎么折成遍数\n\n论文给出一条可直接抄的规则:预算不低于 0.391 个 fp16 遍,就上 6 比特权重,剩下的预算换成 K = floor(B \u002F 0.391) 个顺序,最多四个,每个跑一遍;预算更低就用 4 比特跑一遍。\n\n对照比分是 2 比 7:10 组等成本配对里,「多跑几个顺序」只在 2 组胜出,且都发生在 8 或 6 比特对 fp16 的对比中;「多给几个比特」在两个模型上各赢 7 组。精度-成本前沿上只有 (3 比特, 1 遍)、(4 比特, 1 遍)和 (6 比特, 1 到 4 遍),fp16 和 8 比特一个都没上榜。\n\n反直觉的换算在这儿:从 4 比特出发,平均 2 个顺序相当于单跑 4.7 比特,平均 4 个相当于 5.6 比特。权重从 4 位提到 5.6 位,字节乘 1.37;跑 4 遍要乘 4。\n\n但有一条批处理例外:同一道题的多个顺序若打包一起算,权重只读一次,4 遍几乎免费,这时 4 比特下平均 4 个顺序能捞回 12.6 分,论文说这幅度不亚于其数据里的任何一项增益。作者的结论直接:「能批处理就该平均」。\n\n## 顺手被修好的校准\n\n一个论文没重点宣传但实用的副产品:量化模型不只是变笨,还变得过度自信。0.6B 单次作答的期望校准误差(ECE)在 fp16 下是 0.182,4 比特恶化到 0.332;四顺序平均后掉到 0.032 和 0.087。1.7B 是 0.240、0.284 降到 0.084、0.086。\n\n更妙的是「几个顺序之间是否一致」这个无标签信号,比模型自己的置信度更能指示答得对不对:四序答案一致性的 AUROC,fp16 下 0.6B 是 0.70、1.7B 是 0.80,单次 top probability 是 0.69 和 0.78。到 4 比特,单次 top probability 掉到 0.56,顺序一致性却还有 0.65。\n\n## 这份结论的边界\n\n作者把窟窿列得相当齐。两个模型都来自 Qwen3 一家,0.6B 到 1.7B;Qwen3-4B 没跑,因 fp16 权重约 8 GB,超过了共享 GPU 留出的 6 GB。量化只用最简单的那一种,论文明确不与 GPTQ、AWQ 等具名方法比较。成本单位只数权重字节,忽略激活、KV cache 和反量化开销;论文不报延迟。\n\n还有一处容易漏算的账:tied embedding 和输出矩阵保持 fp16,占 0.6B 权重的 26%、1.7B 的 18%。若引擎每次把整个词表都算 logits,6 比特一遍的真实成本是 0.55 和 0.50,不是 0.391。\n\n样本也偏薄:每任务 300 题,噪声对照只有一两次抽样。更要紧的是脆弱题不跨位宽稳定——fp16 和 4 比特下「哪些题容易翻车」的相关性只有 0.20(0.6B),用 fp16 筛出的脆弱题清单指导量化模型加顺序收益有限。论文提到自己在 Hugging Face Papers 上做标题级检索时没找到把量化和选项顺序放一起量过的先例,也补了一句:这只是单个索引的标题匹配,换了词表述的论文可能存在。\n\n## 所以呢\n\n这篇论文最有用的不是 12.6 这个数,而是把「省内存」和「加算力」从两个互不相干的工程直觉,变成了一张能算的表。对本地跑小模型的人,答案明确:6 比特权重打底,剩余预算换成 4 个顺序取平均;能批处理的话,4 比特加 4 顺序平均更狠。\n\n更深一层的提醒是,4 比特模型不是「变笨了」,是「变成按位置答了」。拿它跑选择题评测,分数带着系统性的槽位偏置——你测的可能不是模型能力,而是它偏爱哪个字母。\n\n本地小模型还剩多少可用空间,答案可能不在于再压几个比特,而在于你愿不愿意为稳定性多付那几遍前向——以及引擎能不能把它们打包成一次权重读取。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14413","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"68b3ad25-26de-4326-bbf4-93ce393f2bbf","en","Bits or Permutations: An Inference Budget Ledger for Models","Intel prices both levers in one unit: 6-bit weights lose no accuracy, 4-bit weights make the model answer by position, 78% of answers in slot A.","Quantizing a small local model to 4 bits and running it several times to average the answers are usually treated as separate levers. The problem is that they interact: quantization changes how sensitive a model is to the order of answer options, and averaging over reorderings is the standard remedy for exactly that. A paper submitted to arXiv on 10 November ([arXiv:2611.14413](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14413)) prices both in the same account, authored by researchers at Intel.\n\n## The setup: two ways to save, four benchmarks\n\nThe models are Qwen3-0.6B and Qwen3-1.7B. The benchmarks are ARC-Easy, ARC-Challenge, MMLU and HellaSwag, 300 items each and 1,200 per model pooled. Quantization is plain round-to-nearest with absmax scales, one fp16 scale per group of 64 weights, at 8, 6, 4 and 3 bits, with fp16 as the reference. Weights are rounded in memory; nothing is packed and no low-bit kernel runs. Embeddings, the tied output matrix and the norms stay in fp16. Every question is asked under eight option orders: the four cyclic shifts of the original order plus the four cyclic shifts of the reversed order.\n\nCost is expressed in one machine-independent unit: weight bytes read times forward passes. A pass costs 0.516 at 8 bits, 0.391 at 6 bits, 0.266 at 4 bits, 0.203 at 3 bits, and 1 at fp16.\n\n## The bit budget: 6 bits is the sweet spot\n\nSix bits loses no accuracy against fp16. It actually scores 1.8 points higher on the 0.6B model and 0.9 on the 1.7B, but the authors say they did not find the reason and make no claim that 6 bits improves a model, so they read the result as \"no loss\". Eight bits costs 0.7 and 0.4 points. Four bits is where it collapses: the 0.6B model loses 14.3 points and the 1.7B loses 5.6. At 3 bits both models sit at chance, 25.4% and 26.6% against a 25% random baseline.\n\nOne 6-bit pass reads 0.391 of the fp16 weight bytes with no accuracy loss, so roughly six tenths of the weight traffic goes away for free.\n\n## At 4 bits the model answers by position\n\nThis is the most interesting finding. Slot skew measures how far the distribution of chosen answer letters deviates from uniform. For the 0.6B model it is 0.15 at fp16 and 0.07 for the 1.7B; at 4 bits the 0.6B jumps to 0.53 and the 1.7B to 0.12. Concretely, the 4-bit 0.6B model puts 78% of its answers in slot A. It is no longer answering the question, it is answering by position. The same-slot share rises from 0.24 to 0.70.\n\nThe consequence: when the correct option happens to sit in the favoured slot, the 4-bit model is right far more often than average, and elsewhere far less often. The range of accuracy across the four gold slots widens from 0.28 at fp16 to 0.76. A benchmark run in the original order averages this effect away, but a user whose own questions have a skewed answer position sees a completely different number.\n\nThe authors also ran a control: Gaussian noise at the same signal-to-noise ratio gives 0.15 and 0.37 on the 0.6B across two draws and 0.19 on the 1.7B across one. The control is inconclusive on whether rounding itself adds slot skew beyond equal-power noise.\n\n## An exchange rate: pricing bits in passes\n\nWith both levers in the same unit, the paper gives a decision rule that can be copied directly. If the budget is at least 0.391 fp16 passes, use 6-bit weights and spend the rest on K = floor(B \u002F 0.391) orders, up to four, one pass each. Below that, use 4 bits and a single order.\n\nThe head-to-head score is 2 to 7. Across 10 cost-matched pairs, extra orders win in only 2 comparisons, always in cells against fp16 at 8 or 6 bits; more bits win in 7 comparisons for each model. The cells on the accuracy-cost frontier are (3 bits, 1 pass), (4 bits, 1 pass) and (6 bits, 1 to 4 passes). Neither fp16 nor 8-bit weights appear on it at all.\n\nThere is also a counter-intuitive conversion. From 4-bit weights, averaging two orders matches the accuracy of a single pass at about 4.7 bits, and averaging four matches about 5.6 bits. Lifting the weights from 4 to 5.6 bits multiplies weight bytes by 1.37, while running four orders multiplies them by 4. While there are still bits to give, bits are the cheaper currency.\n\nThere is one batching exception. When the several orders of one question are processed together, the weights are read once and four orders are nearly free in weight bytes. In that setting, averaging four orders at 4 bits recovers 12.6 points on the 0.6B model, as large as any gain in the paper's data. The authors' conclusion is blunt: a user who can batch should average.\n\n## Calibration gets fixed as a side effect\n\nA byproduct the paper does not advertise heavily is practical. A quantized model is not only less accurate, it is also overconfident. The expected calibration error of a single 0.6B answer is 0.182 at fp16 and degrades to 0.332 at 4 bits; after averaging four orders it drops to 0.032 at fp16 and 0.087 at 4 bits. For the 1.7B the figures are 0.240 and 0.284 falling to 0.084 and 0.086.\n\nBetter still, whether the orders agree with each other is a label-free signal that indicates correctness better than the model's own confidence does. The AUROC of order agreement for the four-order answer is 0.70 on the 0.6B and 0.80 on the 1.7B at fp16, against 0.69 and 0.78 for the single-order top probability. At 4 bits the single-order top probability collapses to 0.56, close to uninformative, while order agreement still holds 0.65. After quantization, the model's own reported certainty is not to be trusted; whether it wavers across orders is the better health check.\n\n## The boundaries of the result\n\nThe authors list their own holes with unusual thoroughness. Both models come from the Qwen3 family, between 0.6B and 1.7B. Qwen3-4B was not run, because its fp16 weights need about 8 GB and the shared GPU had 6 GB free. Only the simplest quantizer was used, and the paper explicitly makes no comparison against named methods such as GPTQ or AWQ, so the 3-bit results are a statement about that simple quantizer. The cost unit counts weight bytes only; it ignores activation traffic, the KV cache, dequantization cost and the fact that prefill is often compute-bound on a GPU. The paper reports no latency at all, only relative units.\n\nOne account is easy to miss: the tied embedding and output matrix stay in fp16 and hold 26% of the 0.6B weights and 18% of the 1.7B. If an engine computes logits over the whole vocabulary on every pass, a 6-bit pass really costs 0.55 and 0.50, not 0.391.\n\nThe sample is also thin: 300 items per task and only one or two noise draws. More importantly, fragile questions do not stay fragile across bit widths. The per-question flip share of fp16 and 4-bit weights correlates at only 0.20 on the 0.6B, so a fragile-question list built at fp16 is a weak guide to where extra permutations would help a quantized model. The authors note that a title-level search on Hugging Face Papers found no earlier study combining quantization with option order, and immediately add that this is a statement about one index and its title-level matching, so a paper using other words may exist.\n\n## So what\n\nWhat is useful here is not the number 12.6 but the conversion of \"save memory\" and \"spend compute\" from two unrelated engineering intuitions into a table you can calculate against. For anyone running a small model locally the answer is fairly clear: start from 6-bit weights and spend the remaining budget on averaging up to four orders, one pass each. If batching is available, 4 bits with four averaged orders is even better.\n\nThe deeper warning is that a 4-bit model does not become dumber, it becomes a model that answers by position. Running multiple-choice evaluation on a 4-bit small model returns a score carrying a systematic slot bias. What gets measured may be the model's preference for a letter rather than its capability, and that bias propagates into downstream filtering, labelling or triage.\n\nHow much room a local small model really has left may therefore not depend on squeezing out another bit, but on whether you are willing to pay a few extra forward passes for stability, and whether your inference engine can pack those passes into a single weight read.","quantization-bits-vs-option-order-budget","2026-11-12T09:10:00Z","2026-11-12T09:18:08.803267Z","2026-11-12T09:18:08.803275Z",true,"agent",247,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"b80697dc-e9ee-40f5-a2c7-3944cd36abba","权重不等长:哈夫曼重编码,边缘LLM再省31.4%","lazyweights-huffman-edge-llm","2026-11-13T23:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a6f5abcc-f1eb-4b6c-ba50-062836f50d0d","量化前先算一遍:二阶筛查排序能信,绝对值不能","quantization-fisher-screen-pre-registered-audit","2026-11-11T03:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"62bc171e-2954-4c08-b3e8-d484acce7894","回放AUC 0.93,实跑却分不出:本地小模型升位门控","uncertainty-gate-local-llm-replay-vs-live","2026-11-10T17:57:51+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"93963f65-e9a2-4058-ba37-563154d1aebd","bf16 状态:推理模型准确率 85.6 掉到 70.8","bf16-state-cache-hybrid-reasoning","2026-11-10T07:15:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"8c0d584e-4bb2-4d00-a5cd-9529824f756d","量化推理模型先管开头30步:CodeQuant2","codequant2-reasoning-model-quantization","2026-11-09T17:10:11+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"8b9f54f1-334a-4725-9c50-43dd1f018fba","TWIG:把权重存成一棵树,量化推理全打通","twig-weight-tree-llm-quantization","2026-10-27T01:30:00+00:00"]