[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llama-cpp-v0-4-0-lazy-tensor-loading":3,"topics-all":38,"news-related-34b087d5-f502-4a46-9599-6f014fe0cf89":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"34b087d5-f502-4a46-9599-6f014fe0cf89","llama.cpp v0.4.0 发布:权重懒加载进主线,小内存也能碰 75B 级 MoE","本地推理引擎 llama.cpp 发布 v0.4.0:张量按需从磁盘读取(--lazy-mode)进主线,配套量化内存上限与逐层 CPU\u002FGPU 路由;新增 Qwen3.8-Flash-Next 与 NVIDIA 75B MoE 初始支持,底层 ggml 升至 0.23.0。","本地推理一直有个老问题:开源模型的参数量一路上涨,用户的内存却不会跟着涨,消费级配置面对 75B 级 MoE 基本没有好选择。llama.cpp 在 9 月 4 日放出的 v0.4.0,主攻的就是这个瓶颈([GitHub release](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Freleases\u002Ftag\u002Fv0.4.0))。\n\n## 权重按需读取:--lazy-mode 改变加载模型的方式\n\n这版最核心的变化是 lazy tensor reading(PR #27794):张量不再一次性全部读进内存,而是推理用到哪块、从磁盘读哪块,运行时开关是 --lazy-mode(#27969)。配套一组内存治理改动:防止模型加载阶段的 RAM 峰值(#27483)、给量化器加内存上限参数 max_buf_size(#27795)、量化过程改为 row-slab 流式处理(#27830)。这套组合拳直接指向小内存跑大权重的场景。\n\n新选项 --n-cpu-ffn(#26622)配合逐层专家路由\u002FFFN(#28323),让 MoE 的前馈部分按层指派到 CPU 或 GPU,混合部署粒度从整个模型细化到了每一层。\n\n## 新架构支持:Qwen3.8-Flash-Next 与 75B 的 Nemotron-3-Puzzle\n\n新模型列表有四个名字:Qwen3.8-Flash-Next(对应新的 qwen4exp 架构,#27742)、NVIDIA Nemotron-3-Puzzle-75B-A9B(#25444)、Nemotron 3.5 的 DSpark 支持(#27804),以及 nanbeige4.2-3B(#27730)。泼盆冷水:Qwen3.8-Flash-Next 目前是 initial support,release notes 明确写着优化尚未完成,qwen4exp 还跟着一串修复(PR #27941、#28123、#28023)。\n\n多模态方向,DeepSeek-V4 视觉输入处理被修复(#28154),新增 DeepSeek-V4-Flash-Vision-Exp 支持(#28133),视频输入参数 --video-* 进了主线(#24318)。\n\n## 底层 ggml 升到 0.23.0:稀疏 attention 与 RDMA\n\n引擎之下,ggml 从 0.22.0 升到 0.23.0,两项新能力最值得注意:稀疏 flash attention(新算子 ggml_flash_attn_ext_set_n_kv_max),已应用在 DeepSeek-V4\u002FGLM 和 qwen4exp 上(#27970);Apple RDMA 作为 RPC 传输层(#26421),配合 RPC 事件与异步后端 API(#18626),多机分摊大模型推理的路径越走越宽。\n\n服务端也更新:per-slot 上下文限制(#24124)让共享同一 server 的请求互不挤占上下文;preserve_reasoning 默认开启(#28174)。\n\n## 所以呢\n\n127k star 的体量,说明在自己机器上跑模型的需求真实且普遍。而最普遍的硬件约束恰恰不是算力,是内存装不下权重——v0.4.0 的懒加载、量化内存上限、逐层路由全部打在这个点上。内存吃紧又想跑新开源 MoE 的用户值得升级;但专为 Qwen3.8-Flash-Next 来的,建议再等等,initial support 和优化到位之间,通常还差几个小版本。","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Freleases\u002Ftag\u002Fv0.4.0","7a64a5d1-9ed6-4f1c-99f9-39b810396a47",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"9fbf1929-3a29-46bc-a9e1-33f92b54a932","en","llama.cpp v0.4.0: lazy tensor loading for memory-tight inference","llama.cpp v0.4.0: on-demand tensor reading, quantizer RAM caps, per-layer CPU\u002FGPU routing, initial Qwen3.8-Flash-Next and 75B Nemotron support.","Local inference has a long-standing problem: open-weight model sizes keep climbing while consumer memory does not, and typical desktops have few good options for 75B-class MoE models. llama.cpp v0.4.0, released September 4, targets exactly this bottleneck ([GitHub release](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Freleases\u002Ftag\u002Fv0.4.0)).\n\n## On-demand tensor reading: --lazy-mode changes how models load\n\nThe headline change is lazy tensor reading (PR #27794): tensors are no longer loaded into memory all at once. The runtime reads each tensor from disk only when inference actually touches it, toggled by the --lazy-mode flag (#27969). This ships with a set of memory-governance changes: preventing RAM peaks during model load (#27483), a new max_buf_size cap for the quantizer (#27795), and row-slab streaming during quantization (#27830). For large MoE models, the combination aims squarely at the \"small memory, big weights\" scenario.\n\nAnother notable option is --n-cpu-ffn (#26622), paired with per-layer expert routing\u002FFFN (#28323): the feed-forward portion of a MoE can now be assigned to CPU or GPU layer by layer — hybrid deployment granularity moves from \"the whole model\" down to \"each layer\".\n\n## New architectures: Qwen3.8-Flash-Next and the 75B Nemotron-3-Puzzle\n\nFour new names appear in the model list: Qwen3.8-Flash-Next (via the new qwen4exp architecture, #27742), NVIDIA's Nemotron-3-Puzzle-75B-A9B (#25444), DSpark support for Nemotron 3.5 (#27804), and nanbeige4.2-3B (#27730). One caveat worth stating plainly: Qwen3.8-Flash-Next is initial support — the release notes explicitly say optimization improvements are still pending, and a string of qwen4exp fixes followed (#27941, #28123, #28023).\n\nOn the multimodal side, DeepSeek-V4 vision input handling was fixed (#28154), DeepSeek-V4-Flash-Vision-Exp support was added (#28133), and video input parameters --video-* landed in mainline (#24318).\n\n## ggml 0.23.0 underneath: sparse attention and RDMA\n\nBelow the engine, ggml moved from 0.22.0 to 0.23.0. Two additions stand out: sparse flash attention (the new operator ggml_flash_attn_ext_set_n_kv_max), already applied to DeepSeek-V4\u002FGLM and qwen4exp (#27970); and Apple RDMA as an RPC transport (#26421), which — together with RPC event and async backend APIs (#18626) — keeps widening the path for spreading a large model across multiple machines.\n\nThe server picked up practical updates too: per-slot context limits (#24124) let concurrent requests sharing one server stop squeezing each other's context, and preserve_reasoning is now enabled by default (#28174).\n\n## So what\n\nThis repository sits at 127k stars — evidence that \"running models on your own machine\" is a real, widespread need. And the most common hardware constraint is precisely not compute, but memory that cannot hold the weights. Lazy loading, the quantizer memory cap, and per-layer routing in v0.4.0 all strike at this exact point. If your machine is memory-tight but you want to run the new open MoE models, this version is worth upgrading for; users coming specifically for Qwen3.8-Flash-Next, however, should wait a bit — between initial support and well-optimized, there are usually a few minor versions of distance.","llama-cpp-v0-4-0-lazy-tensor-loading","2026-09-07T13:10:00Z","2026-09-07T13:09:59.563386Z","2026-09-07T13:09:59.563401Z",true,"agent",236,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"63042acc-9bec-4d6e-960b-8352ed541bb5","bartowski 用 1000 组实验重写 GGUF 量化排布规则","gguf-quant-layout-maps","2026-09-10T21:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"988bbfb8-672a-4c6c-98f7-3a170b6bd8b3","Macaw 把 LFM2.5 装进 1.5GB:4-bit 端侧 LLM 跑 Mac 控制工具链","macaw-lfm25-15gb-edge-mac-agent","2026-08-24T06:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"d5203c93-2022-4769-a6d2-c7765ded2b40","腾讯混元 Hunyuan-A13B 开源实测:80B 总参 \u002F 13B 激活,GQA + FP8\u002FINT4 把 MoE 推理门槛打到消费卡","tencent-hunyuan-a13b-80b-13b-gqa-angelslim-moe","2026-07-30T06:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1abe59b8-844d-4bb1-bc27-cfe28099d101","Anemll\u002FFlash-iOS：把 400B MoE 大模型塞进 iPhone 的端侧实验","anemll-flash-ios-400b-moe-iphone-edge","2026-06-07T12:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"c2e18ecb-00b3-47ee-935a-f0e3cd0dee4a","Holo3.1 把 Computer Use Agent 拉进本地：FP8\u002FNVFP4\u002FQ4 三种量化让消费级 GPU 跑得动","holo3-1-h-company-computer-use-quant","2026-06-07T06:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"8c5bd0e1-551a-405b-b2fc-a67520bf84c5","Colibri v1.11.0 发布:纯 C 引擎直读 DeepSeek V4.1 Flash,552B 从 SSD 流进 CPU","colibri-v1-11-deepseek-v41-flash","2026-09-14T13:09:40+00:00"]