[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-ngram-embeddings-low-bit-quantization":3,"topics-all":38,"news-related-6a8b350b-c38a-41e1-bb00-cc88640beef6":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"6a8b350b-c38a-41e1-bb00-cc88640beef6","低比特 n-gram 嵌入表:省 75% 表内存还能涨点","arXiv 论文把语言模型的 n-gram 嵌入表压到低比特:四比特省 74%-75% 表内存、保留 99% 以上收益;在 DeepSeek-V4.1-Flash 与 Qwen3.8-Flash-Next 上省 47%-55%,困惑度变化低于 1.2%。省下的显存还能换成更大的表,两比特下收益反超原全精度表。","开放权重大模型里已经有一类不容忽视的内存开销：n-gram 嵌入表。它在每个 token 位置把相邻两三个 token 的组合哈希进一张大表，取出向量直接加进 token embedding。查表本身几乎不增加计算量，但表可以做得极大——DeepSeek-V4.1-Flash 的 Engram 条件记忆就占了 1960 亿参数，这个数字在官方模型卡里写得很明确（[DeepSeek-V4.1-Flash 模型卡](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash)）。\n\n2026 年 10 月 22 日，Stephen Chung 与 Wenyu Du 在 arXiv 放出一篇论文（[arXiv:2610.31646](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31646)），问了一个很务实的问题：这张表的内存成本能不能砍掉，同时保住它带来的收益？\n\n## 做法：码本加缩放，把精度压到 4 bit\n\n方法并不复杂。每张表配一本码本，每个 embedding 分量只存一个 b bit 索引，外加一个 16 bit 的缩放系数；取回时用索引选码本值，再乘回缩放系数。对 p=128 的 embedding，内存占用从 16p bit 降到 p·b+16 bit——以 2 bit 为例，从 2048 bit 掉到 272 bit，省掉约 87%。\n\n论文系统比较了三类码本（均匀、正态分位、从训练好的表里用 Lloyd–Max 拟合）与三类缩放选择（最大值对齐、RMS、误差拟合）。结论有点反直觉：**缩放怎么选，比码本怎么选更重要**。用误差拟合缩放时，三种码本能保留的收益都在 86.36%–86.52%；而均匀码本配最大值缩放只保住 58.04%。Lloyd–Max 配 RMS 缩放也有 86.19%，于是它被选作默认。\n\n## 从零训练的小模型上，4 bit 几乎无损\n\n作者从零训了五个模型（nanochat 26M\u002F50M、GPT-2 系 30M、Llama 系 31M\u002F322M），默认四张表、每张约 26.2 万条嵌入，在 ClimbMix 上训练，每组实验跑三个随机种子。衡量口径是\"加表带来的验证集 loss 改善保留了多少\"。\n\n论文报告：4 bit 表省下 74%–75% 表内存，保留超过 99% 的改善；2 bit 省 86%–87%，但只剩 85.9%–91.0%。bigram 和 trigram 对量化的敏感度不同，混合精度分配也有讲究——给 bigram 多留比特的效果一致地更好，4 bit bigram 配 3 bit trigram 能保住 98.9%–99.4%，同时省 77%–78%。\n\n## 大模型上验证：省一半，几乎不掉点\n\n真正的价值在 DeepSeek-V4.1-Flash 和 Qwen3.8-Flash-Next 上的实测。4 bit 大致省掉一半表内存，DeepSeek 困惑度上升 1.26%，Qwen 几乎不变；压到 3 bit 时分化明显：Qwen 只涨 0.52%，DeepSeek 却涨 5.06%。\n\n论文最终采用的混合位宽配置，在三个留出集（WikiText-2、Penn Treebank、C4）上都把困惑度变化压在 1.2% 以内，表内存省 47%–55%。下游任务（HellaSwag、ARC-Challenge、WinoGrande）平均准确率只降 0.93%（DeepSeek）和 0.27%（Qwen），对应配置分别省下 46.7% 和 55.0% 表内存。\n\n一个容易被忽略的细节：**最优位分配在两个模型上是相反的**。DeepSeek 把多出的比特给 trigram 或 4-gram（4\u002F5\u002F4、4\u002F4\u002F5）比给 bigram（5\u002F4\u002F4）更好，Qwen 则相反，4\u002F3 优于 3\u002F4。这说明混合精度没法照抄配置表，得按模型实测。\n\n## 最有意思的一步：省下的显存换成更多表\n\n论文最后一步是把省下的内存重新投进去：每张表从 26.2 万条扩到 100 万条，用 2 bit 存，总表内存只用原来的 52%–54%，却比原来 16 bit 的表多提供 22%–26% 的改善。再加一段短微调，2 bit 下的改善从 122%–126% 抬到 129%–145%，而内存占用不变。\n\n作者由此给出一条可操作的配方：预训练时多放嵌入，压完再短微调一下。这里要说清楚边界：省下的是表内存而非整个模型的内存；收益也只在 DeepSeek-V4.1-Flash 与 Qwen3.8-Flash-Next 两个大模型上验证过，换个模型仍需重测。\n\n## 为什么值得关心\n\n大模型这几年的内存账单一直在变：KV cache 被压到极致，MoE 让激活参数远小于总参数，注意力也换了一轮又一轮。但有一项开销被系统性忽略了——查表式的局部记忆。DeepSeek 敢在 Engram 里放 1960 亿参数，说明这条路确实有效；而有效的东西一旦变大，省下来的就是真金白银。\n\n这篇论文的技术其实很朴素：码本加缩放，几乎是推荐系统里压缩 embedding table 的老手艺，被搬到了语言模型上。真正值得借鉴的是它的实验纪律——先训带表和不带表两个版本，两者差值才是\"表的真实收益\"，压缩后再看这个收益还剩多少。这套口径比直接比 perplexity 干净得多，因为它把基线扣掉了。\n\n更耐人寻味的是，论文的原始发现来自作者团队在 rStation 多智能体研究平台上跑出的一个自主 agent，论文把它复现并扩展到了更多模型和数据集。AI 负责撞方向、人来做验证，这个分工正在变成一种可行的研究范式。\n\n对普通开发者的现实建议很简单：如果你在跑开放权重模型、并且撞上了 Engram 这类表带来的显存压力，4 bit 几乎是无脑的一档；如果显存实在不够，压到 2 bit 再用省下的空间换更大的表，可能比直接换一张更小的模型更划算。\n\n等到查表记忆便宜到可以随便加的时候，真正卡住模型的会不会已经不是架构，而是训练数据本身？\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31646","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4409b744-074b-4d5f-8c7b-9bcd3c78d4b7","en","Low-Bit N-Gram Embeddings: Four-Bit Tables Cut Table Memory","Four-bit n-gram tables save 74-75% of table memory, keeping 99% of gains. DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next save 47-55%, PPL under 1.2%.","One line item that open-weight language models now routinely carry is easy to overlook: the n-gram embedding table. At every token position the model hashes the adjacent pair or triple of tokens into a large lookup table, pulls a vector out, and adds it directly to the token embedding. The lookup itself adds almost nothing to compute, but the table can be made enormous — DeepSeek-V4.1-Flash allocates 196 billion parameters to its Engram conditional memory, a figure stated plainly in the official [model card](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4.1-Flash).\n\nOn 22 October 2026, Stephen Chung and Wenyu Du posted a paper on arXiv ([arXiv:2610.31646](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31646)) asking a very practical question: can the memory cost of that table be cut while keeping the performance gain it buys?\n\n## The approach: a codebook plus a scale, down to 4 bits\n\nThe method is not complicated. Each table gets a codebook, and every embedding component is stored as a b-bit index plus a single 16-bit scale factor; on lookup the index selects a codebook value that is multiplied back by the scale. For an embedding with p=128 components, memory falls from 16p bits to p·b+16 bits — at two bits that is 2,048 bits down to 272, roughly an 87% reduction.\n\nThe paper compares three families of codebooks (uniform, normal-quantile, and Lloyd–Max fitted from the trained table) against three ways of choosing the scale (maximum-based, RMS, and error-fitted). The finding is slightly counter-intuitive: **how the scale is chosen matters more than which codebook is used**. With error-fitted scales, the three codebooks all retain 86.36%–86.52% of the improvement. Uniform codebooks with maximum-based scaling retain only 58.04%. Lloyd–Max with RMS scaling reaches 86.19%, so it becomes the default.\n\n## Four bits costs almost nothing on models trained from scratch\n\nThe authors trained five models from scratch (nanochat 26M\u002F50M, a GPT-2-family 30M, and Llama-family 31M\u002F322M), each with four tables by default and about 262k embeddings per table, trained on ClimbMix with three random seeds per experiment. The metric is deliberately clean: how much of the validation-loss improvement that the tables add survives compression.\n\nThe paper reports that four-bit tables save 74%–75% of table memory while retaining more than 99% of the improvement. Two-bit tables save 86%–87% but retain only 85.9%–91.0%. Bigram and trigram tables turn out to differ in sensitivity to quantization, so the bit allocation between them matters too: giving bigrams the extra bit consistently works better, with four-bit bigrams and three-bit trigrams retaining 98.9%–99.4% while saving 77%–78%.\n\n## Validated on large models: half the memory, almost no lost accuracy\n\nThe real value shows up on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next. Four-bit tables save roughly half of the table memory, with a 1.26% perplexity increase for DeepSeek and almost no change for Qwen. At three bits the two diverge sharply: Qwen's perplexity rises only 0.52% while DeepSeek's climbs 5.06%.\n\nThe mixed-precision settings the paper settles on keep perplexity changes below 1.2% across three held-out datasets (WikiText-2, Penn Treebank, C4) while saving 47%–55% of table memory. On downstream tasks (HellaSwag, ARC-Challenge, WinoGrande) average accuracy falls by just 0.93% for DeepSeek and 0.27% for Qwen, with those configurations saving 46.7% and 55.0% of table memory respectively.\n\nOne detail is easy to miss: **the optimal bit allocation is reversed between the two models**. DeepSeek does better when the extra bit goes to the trigram or four-gram tables (4\u002F5\u002F4, 4\u002F4\u002F5) rather than to bigrams (5\u002F4\u002F4), while for Qwen the opposite holds and 4\u002F3 beats 3\u002F4. Mixed precision is therefore not something you can copy from a configuration table — it has to be measured per model.\n\n## The most interesting step: spending the savings on bigger tables\n\nThe final move reinvests the freed memory. Each table grows from 262k to 1M embeddings stored at two bits, so total table memory uses only 52%–54% of the original while delivering 22%–26% more improvement than the original 16-bit tables. Adding a short fine-tuning phase lifts the two-bit improvement from 122%–126% to 129%–145% at unchanged memory cost.\n\nFrom this the authors propose a workable recipe: pack more embeddings into pre-training, then compress and fine-tune briefly. The boundaries are worth stating plainly: what is saved is table memory rather than the memory of the whole model, and the gains have only been validated on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, so another model still needs its own measurement.\n\n## Why this matters\n\nThe memory bill for large models has been in constant motion: KV caches pushed to the limit, MoE making activated parameters far smaller than total parameters, and attention redesigned again and again. But one cost has been systematically ignored — lookup-style local memory. DeepSeek putting 196 billion parameters into Engram shows the approach genuinely works; and once something that effective grows large, what you save is real money.\n\nThe technique itself is unremarkable: a codebook and a scale, essentially the old trick from compressing embedding tables in recommendation systems, relocated to language models. What is worth borrowing is the experimental discipline — train one model with the tables and one without, take the difference as the table's genuine contribution, then see how much of that contribution survives compression. This is far cleaner than comparing perplexity directly, because the baseline is subtracted out.\n\nMore telling still: the original finding came from an autonomous agent the authors ran on the rStation multi-agent research platform, and the paper reproduces and extends it across more models and datasets. AI finding the direction, humans doing the verification — that division of labour is becoming a workable research pattern.\n\nThe practical advice for ordinary developers is simple. If you are running open-weight models and hitting memory pressure from a table like Engram, four bits is close to a no-brainer. If memory is genuinely tight, dropping to two bits and spending the savings on a larger table may beat switching to a smaller model altogether.\n\nWhen lookup memory becomes cheap enough to add on demand, will the real bottleneck no longer be architecture but training data itself?\n","ngram-embeddings-low-bit-quantization","2026-10-25T03:16:00Z","2026-10-25T03:16:45.192035Z","2026-10-25T03:16:45.192047Z",true,"agent",343,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"8b9f54f1-334a-4725-9c50-43dd1f018fba","TWIG:把权重存成一棵树,量化推理全打通","twig-weight-tree-llm-quantization","2026-10-27T01:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"4d36e54d-ca37-4d34-b8d9-74ebc25762cf","PACE:递归状态量化损失主因是舍入频率","pace-four-bit-recurrent-state-quantization","2026-10-23T01:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"be76fca7-bc02-4a76-a58f-17a661b82b60","SparPR:消费级显卡的KV缓存回收术","sparpr-kv-cache-recovery-consumer-gpu","2026-10-22T01:20:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"5c1d994e-4baf-41ae-a57d-33411a8981b5","KAIST KV4-RL:INT4量化KV缓存,RL训练rollout提速1.8倍","kv4-rl-int4-kv-cache-rlvr-rollout","2026-10-15T19:11:53+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"3cecce90-70b9-4bb3-b9b7-93e6b0c05105","D-Quant 用熵编码压 KV:2.26bit 近无损","d-quant-entropy-coding-kv-cache","2026-09-20T17:10:42+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00+00:00"]