[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-warp-engine-glm-flash-nvme-inference":3,"topics-all":38,"news-related-1a50eda4-e62d-40ba-8f9d-dab756067e2d":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"1a50eda4-e62d-40ba-8f9d-dab756067e2d","16GB 内存跑 313B GLM-5.3-Flash:WARP 把专家权重搬进 NVMe","开源 C 推理引擎 WARP 把 MoE 主干留在内存、激活专家从 NVMe 流式读取:GLM-5.3-Flash 最低 5.14GB 内存即可运行,16GB 笔记本实测 3.06 tok\u002Fs,约为 64GB 机器的九成;2.78 万亿参数的 Kimi K3 也能在 64GB MacBook 上完整跑通。","313B 参数的 fp8 权重要占 328GB 磁盘,而一台 16GB 内存的笔记本连零头都装不下——这是所有\"本地跑大模型\"方案面对的第一堵墙。常规答案无非三条:暴力量化、蒸馏小模型、或者干脆交给云端。SQLite Cloud 开源的 WARP 引擎(前名 WASTE)给出了第四条路:权重大部分住 NVMe,内存只留缓存。\n\n## 把专家流进 NVMe,而不是塞进内存\n\nWARP 是一个纯 C 编写、推理路径只依赖 libc 和 pthreads 的嵌入式引擎,Apache 2.0 协议,GitHub 已有 2.3k star。它的思路针对 MoE 架构量身定做:共享主干(trunk)常驻内存,每个 token 实际激活的专家按需从磁盘流式读取,剩余内存全部充当有界专家缓存。容器格式经过精心排列,一个专家恰好对应一次对齐读;前瞻路由器还会预测下一层要用的专家、提前发起 IO,与计算重叠。\n\n量化策略是分层的:专家用 3-bit 残差矢量量化,更敏感的共享权重保留 4\u002F8-bit。再加上 Kimi K3 的线性注意力与压缩 latent KV cache——4K 上下文时 KV 缓存只要约 0.21GB 而非 11.25GB——打开整个 K3 只需 29.19GB 内存。\n\n## 实测:数字与翻车点\n\n项目在 64GB MacBook Pro(M5 Pro、内置 SSD)上的测量:\n\n- **GLM-5.3-Flash(313.89B 总参、17.31B 激活)**:最低 5.14GB 内存可跑,实测 3.32 tok\u002Fs(64 token)\u002F3.86 tok\u002Fs(200 token);16GB 内存机器自动降到 3.06 tok\u002Fs,约为 64GB 机器的九成\n- **Kimi K3(2.78 万亿参数)**:完整模型、非蒸馏非剪枝,0.45–0.62 tok\u002Fs,一个冷 token 要读约 17GB 专家\n- **Kimi-Linear 48B**:17.22 tok\u002Fs,最低 1.32GB 内存\n\nREADME 里最诚实的是两类\"负结果\":专家缓存不是越大越好——超过预算后命中率仍在涨、吞吐却暴跌八倍,因为 cache hit 变成了 page fault;每 token 专家数从 16 降到 8 可以换 1.49 倍速度(KL 散度 0.037),降到 4 则模型直接跑飞。存储才是主约束:内置 SSD 12.78GB\u002Fs,测试用的 USB 硬盘柜只有 0.94GB\u002Fs。\n\n## 几个值得记住的细节\n\n- 整个项目\"人类主导、LLM 写码\",作者说这是迭代算法假设速度的唯一方式,终极目标是让 K3 在本地改进引擎本身\n- 所有层对着 PyTorch 参考实现校验,GLM 最终 logits 相对 L2 误差 2.41e-5,argmax 与 top-10 完全一致\n- 0.6.8 起支持 DeepSeek V3\u002FR1\u002FKimi K2 家族转换;GLM-5.3-Flash 的视觉塔(282MB)按需加载,一张 200×140 图只占 40 个 token 位置\n\n对\"本地能不能跑大 MoE\"这个问题,WARP 给出的答案是把瓶颈从 RAM 挪到了 NVMe 带宽——而后者每台笔记本都有。项目名字原本就叫 WASTE,作者的注脚是:每个云端 token 都付两次钱,一次在账单上,一次在数据中心电费里。当开源权重越来越大方、SSD 越来越快,\"桌上这台机器其实跑得动\"正在从安慰变成工程事实。详见 [GitHub 仓库](https:\u002F\u002Fgithub.com\u002Fsqliteai\u002Fwarp)。","https:\u002F\u002Fgithub.com\u002Fsqliteai\u002Fwarp","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"fca9258a-9430-455a-b95d-b9fae5e373a8","ai-inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"132db908-ffc4-4822-af08-dfd7d4a961a7","en","Run 313B GLM-5.3-Flash in 16GB of RAM: WARP moves expert weights into NVMe","Open-source C inference engine WARP keeps the MoE trunk in RAM and streams activated experts from NVMe: GLM-5.3-Flash runs in as little as 5.14GB of memory, with a 16GB laptop measured at 3.06 tok\u002Fs — about 90% of a 64GB machine; the full 2.78-trillion-parameter Kimi K3 also runs on a 64GB MacBook.","A 313B-parameter model in fp8 occupies 328GB of disk, while a 16GB laptop cannot even hold a fraction of it — the first wall every \"run big models locally\"方案 runs into. The usual answers are brute-force quantization, distilled small models, or just giving up and calling the cloud. WARP (formerly WASTE), an open-source engine from SQLite Cloud, offers a fourth path: let the weights live mostly on NVMe and keep RAM as a cache.\n\n## Streaming experts from NVMe instead of stuffing them into RAM\n\nWARP is an embeddable inference engine written in pure C, with the inference path depending only on libc and pthreads, under Apache 2.0, and already at 2.3k stars on GitHub. The design is tailored to MoE architectures: the shared trunk stays resident in memory, the experts actually activated by each token are streamed from disk on demand, and all remaining RAM serves as a bounded expert cache. The container format is arranged so that one expert corresponds to exactly one aligned read; a lookahead router predicts the experts needed by the next layer and starts reading early, overlapping IO with compute.\n\nQuantization is tiered: experts use 3-bit residual vector quantization while the more sensitive shared weights stay at 4\u002F8 bits. Combined with Kimi K3's linear attention and compressed latent KV cache — about 0.21GB at 4K context instead of 11.25GB — opening the entire K3 requires only 29.19GB of memory.\n\n## Measured: the numbers and the failure modes\n\nMeasured on a 64GB MacBook Pro (M5 Pro, internal SSD):\n\n- **GLM-5.3-Flash (313.89B total, 17.31B active)**: runs with as little as 5.14GB of RAM, measured at 3.32 tok\u002Fs over 64 tokens and 3.86 tok\u002Fs over 200 tokens; a 16GB machine automatically drops to 3.06 tok\u002Fs, about 90% of the 64GB figure\n- **Kimi K3 (2.78 trillion parameters)**: the full model, not distilled or pruned, at 0.45–0.62 tok\u002Fs; a cold token reads about 17GB of experts\n- **Kimi-Linear 48B**: 17.22 tok\u002Fs with a 1.32GB memory floor\n\nThe most honest parts of the README are the negative results: a bigger expert cache is not always better — past the budget the hit rate keeps climbing while throughput collapses eightfold, because a cache hit becomes a page fault; and dropping experts per token from 16 to 8 buys a 1.49x speedup (KL divergence 0.037), while dropping to 4 breaks the model outright. Storage is the main constraint: the internal SSD sustains 12.78GB\u002Fs versus 0.94GB\u002Fs on the tested USB enclosure.\n\n## Details worth remembering\n\n- The whole project is \"human-driven, LLM-written code\" — the author says this is the only way to iterate on algorithmic hypotheses fast enough, with the ultimate goal of having K3 improve the engine itself locally\n- All layers are checked against a PyTorch reference; GLM's final logits agree within relative L2 of 2.41e-5, with identical argmax and top-10\n- Since 0.6.8 the converter handles the DeepSeek V3\u002FR1\u002FKimi K2 family; GLM-5.3-Flash's vision tower (282MB) loads on demand, and a 200×140 image costs only 40 token positions\n\nOn the question of whether local hardware can run large MoE models, WARP's answer is to move the bottleneck from RAM to NVMe bandwidth — and every laptop has that. The project was originally named WASTE, with the author's footnote: every cloud token is paid for twice, once on the invoice and once in datacenter electricity. As open weights keep getting more generous and SSDs keep getting faster, \"the machine on your desk can actually run this\" is turning from consolation into engineering fact. See the [GitHub repository](https:\u002F\u002Fgithub.com\u002Fsqliteai\u002Fwarp).","warp-engine-glm-flash-nvme-inference","2026-08-31T13:00:00Z","2026-08-30T19:07:16.743909Z","2026-08-30T19:07:16.743917Z",true,"agent",207,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"68072ee1-fc37-4064-ab18-09550ae72d1b","GLM-5.3-Flash 把 320B MoE 跑在国产芯片上:Flash 价位和 $0.15 API 的混合注意力栈","glm-5-3-flash-chinese-chips-hybrid-attention","2026-08-27T03:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"21fe3c11-4ba4-4801-b6fc-60c4ae559dc1","Yandex 逆流开源:35B 参数的 T5 MoE,每个 token 只激活 0.6B","yandex-aliceai-t5-sparse-moe","2026-09-16T19:11:43+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"cb1e799d-d6d7-4ab9-9eaf-bea0aa432b06","Mistral 模型进驻 Firefox:119B 开放权重模型驱动浏览器 AI 助手","mistral-small-4-firefox-smart-window","2026-09-16T17:07:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"8e730a3d-439b-45cf-961d-f77cf01469fd","Cohere 开源 218B 翻译专用 MoE:25B 激活,自测评分超 DeepL,2×H100 可部署","cohere-north-small-translate","2026-09-11T19:07:20+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"42b7939c-1b44-43b8-95cf-a8fc2204560d","NVIDIA 开源 Personal AI Router，把家里 RTX 与 Mac 拼成本地 AI 集群","nvidia-personal-ai-router-pair-beta","2026-09-04T03:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"d941056b-c2e7-42e5-965a-a982c20b1169","Qwen3.8-Flash-Next 架构细节:Gated Residual 多分支残差 + QSA micro-block 稀疏注意力","qwen3-8-flash-next-cost-efficiency-architecture","2026-09-02T02:00:00+00:00"]