[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-lazyweights-huffman-edge-llm":3,"topics-all":38,"news-related-b80697dc-e9ee-40f5-a2c7-3944cd36abba":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"b80697dc-e9ee-40f5-a2c7-3944cd36abba","权重不等长:哈夫曼重编码,边缘LLM再省31.4%","UT Austin与凯斯西储团队提出LazyWeights:llama.cpp格式与GGUF文件不变,GPTQ把整数码成形得更可压缩,哈夫曼无损编码,DRAM与L2之间0.21mm²的硬件单元按需解码。实测同困惑度下比llama.cpp最优格式小5.7–31.4%,4GB板流式跑Mistral-7B快6.6倍。","4GB的Jetson Nano卡在容量,8GB的Orin Nano卡在带宽:7B模型fp16要14GB,就算压到4bit,每生成一个token还是要把数GB权重从内存搬一遍。而目前的主流量化格式——GPTQ、AWQ、llama.cpp的K系和I系——都在做同一件浪费的事:给矩阵里每个整数码分配同样多的位数。权重分布是钟形的,中间的码出现得远比两端频繁,4bit字母表的熵明显低于4bit,这块空间一直没被拿走。[LazyWeights](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17108)(arXiv 2611.17108,来自UT Austin与凯斯西储大学团队)就是来吃掉这块空间的。\n\n## 三步:成形、编码、按需解码\n\nLazyWeights不另起炉灶:文件仍是合法GGUF,原版llama.cpp直接加载,质量完全由量化器决定。它只动三个环节。\n\n**成形**:在GPTQ里加两个旋钮。一是让子块共享步长(参数γ),不再按子块各自拉满量程,把码赶到高频区;二是把步长整体放大(×1、×1.25、×1.5按点选),熵随步长下降,误差交给GPTQ的误差反馈吸收。\n\n**编码**:每个矩阵建一棵限长11位的规范哈夫曼码,64×256的tile拆成32条交织流并行解码,4bit格式两个码拼成一个对符号、一步解两个权重。含全部开销后距零阶熵只有1.2–2.1%。\n\n**解码**:关键是位置。DRAM与L2之间放一个约0.21mm²、1GHz、7nm的解码单元:DRAM只搬编码后的字节,L2收到定宽码,GPU kernel一行不改。为什么必须上硬件?论文实测软件解码比DRAM速率慢15–20倍,同样的码在定宽下跑反而快2.5–3.2倍——解码速度不解决,压缩就是负资产。\n\n## 论文实测的数字\n\n- 同困惑度预算(≤1%与≤3.5%,WikiText-2)下,编码后的Phi-3-mini、Mistral-7B、SmolLM-1.7B比同预算最优的llama.cpp格式小5.7–31.4%;上限出现在Mistral-7B≤1%档,5.668降到3.886 bit\u002F权重。哈夫曼本身贡献6.2–20.7%,其余来自量化器成形。\n- Orin上贪心解码每token时间、能耗最多降36%、38%;叠加提示查找投机(草稿来自n-gram表,无草稿模型)后到54%、44%。\n- 4GB Nano上,常驻模型比CPU版llama.cpp每token少58–64%时间、53–57%能耗;放不下的Mistral-7B改流式,3.89 bit\u002F权重下比llama.cpp快6.6倍——这一档还是纯软件解码。\n- 工程可信度给得足:C模型复现17个bundle零失配,Verilator对RTL协同仿真29\u002F29矩阵位级一致,四引擎在AMD ZCU102 FPGA(112.5MHz)上把4.84亿权重全部位精确解出;对比rANS,哈夫曼速率从不更差,同吞吐下rANS配对符号要多4.0倍面积。\n- 下游任务:HellaSwag、ARC-e、PIQA最多比基线低1.5点,GSM8K最多低3.2点(Mistral≤3.5%档)。\n\n## 论文自己画出的边界\n\n这段比数字更值得读。第一,解码单元的时延与能耗是用实测器件数据加预布局综合推算的,不是流片结果;FPGA验证的是位精确,但跑在FPGA时钟上。第二,没测Llama-3和Qwen3——论文引用他人结论说这两家量化表现更差;长上下文里fp16 KV cache会稀释收益,4K token时每pass 0.54GB,对着3.5GB权重。第三,步长放大非单调:Mistral放大到×1.75困惑度反而+15.5%,所以每个模型都要单独搜一遍,没有万能配方。\n\n## 所以呢\n\n论文的结论句可以直接抄:定宽是承诺,不是必然。作者把全套都放出来了——MIT协议的代码、设备日志、六个工作点的权重(Hugging Face),GGUF文件直接进原版llama.cpp。对做边缘部署的人来说,「放得下\u002F放不下」这条线是可以用信息论挪动的;论文在相关工作里还点明:BitLLM那类省每码计算的路数,与这里省每码字节数正交,可以叠加。\n\n真正的问题在落地:0.21mm²不大,但它要长在内存控制器旁边,得芯片厂商点头。你愿意为多装一个7B模型,等一个解码单元进下一代边缘SoC吗?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17108","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"0f1d073b-792a-4809-a826-35952531ca94","en","LazyWeights: Huffman-Coded Edge LLMs Shrink by 31.4%","Huffman-coded weights decoded on demand by a 0.21 mm2 DRAM-L2 unit: 5.7-31.4% smaller at equal perplexity, 6.6x faster Mistral-7B on 4 GB.","A 4 GB Jetson Nano is capacity-bound; an 8 GB Orin Nano Super is bandwidth-bound. A 7B model needs 14 GB in fp16, and even at 4 bits each token moves gigabytes through a phone-class memory system. Yet every deployed quantization format - GPTQ, AWQ, llama.cpp's K- and I-quants - spends equal bits on every integer code of a matrix, although weight codes are bell-shaped: inner codes dominate, and the entropy of a 4-bit alphabet sits well below 4 bits. [LazyWeights](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17108) (arXiv 2611.17108, from UT Austin and Case Western Reserve University) is built to claim exactly that slack.\n\n## Three moves: shape, code, decode on demand\n\nLazyWeights does not invent a new container: files remain valid GGUF that unmodified llama.cpp loads, and quality is the quantizer's alone. It touches three stages instead.\n\n**Shape.** Two knobs inside GPTQ. Shared sub-block steps (parameter gamma) stop each sub-block from stretching its codes over the whole alphabet, concentrating codes where the weights are; a step scale (x1, x1.25, x1.5, chosen per point) lowers entropy as the grid coarsens, with GPTQ's error feedback absorbing the damage.\n\n**Code.** One canonical Huffman code per matrix, length-limited to 11 bits; each 64x256 tile becomes 32 interleaved lane streams decoded in parallel; in 4-bit formats two codes form one pair symbol, two weights per decode step. All overheads included, the bundle stays within 1.2-2.1% of zeroth-order entropy.\n\n**Decode.** Location is the trick. A roughly 0.21 mm2 decoder unit (1 GHz, 7 nm) sits between DRAM and L2: DRAM moves only coded bytes, L2 receives fixed-width codes, GPU kernels run unchanged. Why hardware? The paper measures software decoders falling 15-20x short of DRAM rate on the Orin, making a token 2.5-3.2x slower than the same codes at fixed width - until decoding speed is solved, compression is a liability.\n\n## What the paper measured\n\n- At equal perplexity budgets (within 1% and 3.5% on WikiText-2), coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best llama.cpp format in the same budget; the ceiling is Mistral-7B within 1%, from 5.668 down to 3.886 bits per weight. Huffman coding alone contributes 6.2-20.7%; the quantizer supplies the rest.\n- On the Orin, greedy decoding cuts per-token time and energy by up to 36% and 38%; with prompt-lookup speculation (drafts from an n-gram table, no drafter model) that rises to 54% and 44%.\n- On the 4 GB Nano, resident models need 58-64% less time and 53-57% less energy per token than CPU llama.cpp; Mistral-7B, which does not fit, streams at 3.89 bits per weight and runs 6.6x faster than llama.cpp - still with software decoding.\n- Engineering evidence is solid: a C model reproduces 17 bundles with zero mismatches, Verilator co-simulation matches the RTL bit- and cycle-exactly on 29\u002F29 matrices, and four engines on an AMD ZCU102 FPGA (112.5 MHz) decode 484M weights bit-exactly. Against rANS, Huffman is never larger in rate, and rANS needs 4.0x the area for pair symbols at equal throughput.\n- Downstream: HellaSwag, ARC-easy and PIQA sit at most 1.5 points below baseline; GSM8K at most 3.2 (Mistral within 3.5%).\n\n## The limits the authors draw themselves\n\nThis section matters more than the headline numbers. First, the unit's latency and energy are projections from measured device numbers plus pre-layout synthesis - not a fabricated SoC; the FPGA proves bit-exactness at FPGA clocks. Second, Llama-3 and Qwen3 were not evaluated - the paper relays prior findings that they quantize worse - and long contexts dilute the saving, with an fp16 KV cache of 0.54 GB per pass against 3.5 GB of weights at 4K tokens. Third, the step scale is non-monotone: at x1.75 Mistral perplexity jumps 15.5%, so every model needs its own search; there is no universal recipe.\n\n## So what\n\nThe closing line is worth quoting: fixed width is a commitment, not a necessity. The authors released everything - MIT-licensed code, device logs, and the six operating points on Hugging Face, with GGUF files that drop into unmodified llama.cpp. For edge deployment, the line between what fits and what does not can be moved with information theory, and the paper notes in related work that approaches like BitLLM, which cut compute per code, are orthogonal to cutting bytes per code and compose with it.\n\nThe real question is adoption: 0.21 mm2 is small, but it must sit next to the memory controller, which takes a chip vendor's blessing. Would you wait one silicon generation to fit a 7B model on your edge board?","lazyweights-huffman-edge-llm","2026-11-13T23:30:00Z","2026-11-13T23:08:53.308324Z","2026-11-13T23:08:53.308333Z",true,"agent",240,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"081148f2-39eb-432d-8841-5cb1e58ed527","省内存还是多跑几遍:本地小模型怎么花推理预算才划算","quantization-bits-vs-option-order-budget","2026-11-12T09:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a6f5abcc-f1eb-4b6c-ba50-062836f50d0d","量化前先算一遍:二阶筛查排序能信,绝对值不能","quantization-fisher-screen-pre-registered-audit","2026-11-11T03:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"62bc171e-2954-4c08-b3e8-d484acce7894","回放AUC 0.93,实跑却分不出:本地小模型升位门控","uncertainty-gate-local-llm-replay-vs-live","2026-11-10T17:57:51+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"93963f65-e9a2-4058-ba37-563154d1aebd","bf16 状态:推理模型准确率 85.6 掉到 70.8","bf16-state-cache-hybrid-reasoning","2026-11-10T07:15:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"8c0d584e-4bb2-4d00-a5cd-9529824f756d","量化推理模型先管开头30步:CodeQuant2","codequant2-reasoning-model-quantization","2026-11-09T17:10:11+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"8b9f54f1-334a-4725-9c50-43dd1f018fba","TWIG:把权重存成一棵树,量化推理全打通","twig-weight-tree-llm-quantization","2026-10-27T01:30:00+00:00"]