[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-janas-ssd-streaming-moe-inference":3,"topics-all":38,"news-related-ad9f5296-75bf-434c-a527-dc2703eb8c6a":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"ad9f5296-75bf-434c-a527-dc2703eb8c6a","Janas 引擎:80B 模型塞进 32GB 内存笔记本","纯 C 引擎 Janas 把模型权重留在磁盘,推理时只把命中的 MoE 专家从 NVMe SSD 流式读入,热专家常驻内存缓存。实测 32GB 内存笔记本跑 Qwen3-Next-80B,写入 37 tok\u002Fs,同机 13 模型对比全面快过 llama.cpp;作者自曝是实验品,仅在一台机器验证。","在 32GB 内存的笔记本上跑一个 48.4GB 的 80B 参数模型,还能以 37 tok\u002Fs 的速度往外吐字——这不是配置清单上的幻想,而是开源项目 Janas 本周晒出的实测成绩。\n\n## 把\"装不下\"变成\"读得快\"\n\nJanas-LLM 是一个用纯 C 写成的推理引擎,除 C 标准库外零依赖,GPL-3.0 开源。它的核心思路不是压缩模型,而是换问题:权重整体留在磁盘上,推理时每个 token 只把路由命中的那几个 MoE 专家从 NVMe SSD 流式读进内存,反复被调用的热专家常驻缓存,冷专家用完即走。内存只需要装下\"活跃工作集\",而不是整个模型。\n\n作者 Maurizio \"camauri\" Cammalleri 的实测机器是一台 Intel Core Ultra 9 185H 笔记本:32GB DDR5、1TB NVMe(读速约 5.7GB\u002Fs)、集成 Arc GPU 走 Vulkan。跑 Qwen3-Next-80B-A3B 时,热专家缓存约占 19GB,其余全从盘上读。\n\n## 数字:37 对 2.8,以及 13 连胜\n\n在 2026 年 10 月 9-12 日的同机对比里,Janas-LLM 读 1024 token 提示 62 tok\u002Fs,llama.cpp 为 16;写入 24 tok\u002Fs 对 2.8 tok\u002Fs;叠加投机解码后聊天场景达到 37 tok\u002Fs。作者称在其测试机上,13 个模型(覆盖 Qwen3、Qwen3.5、Qwen3.6、Qwen3-Next 与 Gemma 4 的 dense\u002FMoE 版本,Q4_K_M 量化)对比中 Janas-LLM 全部更快——包括 Qwen3.5-0.8B 这类小模型(写入 109 对 32 tok\u002Fs)。\n\n80B 这一档的对照最直观:llama.cpp 常规构建在 GPU 参与时直接内存耗尽——CPU 后端要拷贝权重重排,Vulkan 后端又留下系统无法回收的缓冲;关掉重排、改用 mmap 才跑起来,写入只有 2.8 tok\u002Fs。Janas 无需任何选项就跑完了全程。\n\n也有例外:unsloth 的 3-bit Qwen3-30B 读提示反而比 llama.cpp 慢(100 对 112 tok\u002Fs),因为该量化格式的权重还没走上 GPU 快速路径——README 里写得明明白白。投机解码同样有边界:Qwen 大模型上草稿存活率 74-89%,换到意大利语文本,Qwen3.5-2B 只活下来约三分之一,收益接近零。\n\n## 一人项目,和一句\"这是实验品\"\n\nREADME 开头就是一盆冷水:\"This is an experiment, not a product\"。公开 API 和命令行每周变好几次;引擎故意吃满空闲内存、用 O_DIRECT 全速读盘,机器余量不足时可能把桌面推进 swap,作者自述 GPU 曾引发一次驱动重置带走整个桌面;全部开发与测量只在这一台笔记本上完成,其他 CPU、独显、慢盘均未验证。\n\n但它同时展示了 MoE 时代端侧推理的一条真实路线:稀疏激活意味着\"模型大小\"不再等于\"内存需求\",瓶颈从容量迁移到带宽与缓存命中率——路由若老在热门专家上打转,速度就稳;一旦游走,磁盘就是天花板。这与此前 Colibri 从 SSD 流式跑 552B 模型、伯克利把 KV 缓存压进 SSD 是同一方向的不同切面。\n\n对普通用户,现在的建议是别急着上生产;对关注本地推理的人,这个数百 star 的仓库值得盯——一人、纯 C、零依赖做到这个完成度,本身就是对\"端侧跑大模型必须堆显存\"的一次反证。\n\n参考: https:\u002F\u002Fgithub.com\u002Fprabanta-dev\u002Fjanas","https:\u002F\u002Fgithub.com\u002Fprabanta-dev\u002Fjanas","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"e0e39ce2-c05b-4eb1-af0f-eb1891891665","en","Janas: a pure-C engine running 80B models in 32GB RAM","Author-reported: pure-C engine Janas streams MoE experts from NVMe SSD, running an 80B model on a 32 GB laptop at 37 tok\u002Fs where llama.cpp hits 2.8.","Running an 80-billion-parameter model — a 48.4 GB file — on a laptop with 32 GB of RAM, at 37 tokens per second, is the headline number an open-source project called Janas published this week.\n\n## Turning \"won't fit\" into \"reads fast enough\"\n\nJanas-LLM is an inference engine written in pure C with no dependencies beyond the C library, released under GPL-3.0. Its core idea is not to shrink the model but to change the question: all weights stay on disk, and during inference each token streams only the few MoE experts its router activates from an NVMe SSD. Experts that keep being hit stay in a memory cache; cold ones are read and released. RAM only has to hold the active working set, not the whole model.\n\nThe author, Maurizio \"camauri\" Cammalleri, built and measured everything on one laptop: Intel Core Ultra 9 185H, 32 GB DDR5, a 1 TB NVMe SSD reading at about 5.7 GB\u002Fs, and the integrated Arc GPU through Vulkan. Running Qwen3-Next-80B-A3B, the hot-expert cache takes about 19 GB; the rest is read from disk on demand.\n\n## The numbers: 37 vs 2.8, and 13 straight wins\n\nIn same-machine comparisons measured on 9-12 October 2026, Janas-LLM read a 1,024-token prompt at 62 tok\u002Fs against llama.cpp's 16, wrote at 24 tok\u002Fs against 2.8, and reached 37 tok\u002Fs in chat with speculative decoding. The author reports that on his machine Janas-LLM was faster than llama.cpp on all 13 models compared in Q4_K_M — spanning dense and MoE builds of the Qwen3, Qwen3.5, Qwen3.6, Qwen3-Next and Gemma 4 families — including small ones like Qwen3.5-0.8B (109 vs 32 tok\u002Fs writing).\n\nThe 80B case is the sharpest contrast: llama.cpp's usual build ran out of memory as soon as the GPU took part — the CPU backend copies weights to repack them, and the Vulkan backend leaves buffers the system cannot reclaim. Built without repacking and reading through mmap, it managed 2.8 tok\u002Fs writing. Janas needed no options at all.\n\nThere are honest exceptions. unsloth's 3-bit Qwen3-30B reads a prompt slower than llama.cpp (100 vs 112 tok\u002Fs) because those weights do not yet take the GPU fast path — the README says so plainly. Speculative decoding has boundaries too: drafts survived 74-89% on the larger Qwen models, but on Italian prose Qwen3.5-2B kept only about a third and gained almost nothing.\n\n## A one-person project, and \"this is an experiment\"\n\nThe README opens with cold water: \"This is an experiment, not a product.\" The public API and command line change several times a week; the engine deliberately fills free memory and reads the disk flat-out with O_DIRECT, which can push a tight machine into swap, and one GPU session triggered a driver reset that took the desktop down. Everything was developed and measured on that single laptop; other CPUs, discrete GPUs and slower disks are untested.\n\nYet it demonstrates a real route for local inference in the MoE era: sparse activation means model size no longer equals memory need — the bottleneck moves from capacity to bandwidth and cache hit rate. If routing keeps visiting the hot experts, speed holds; once it wanders, the disk becomes the ceiling. It is a different cut of the same direction as Colibri streaming a 552B model from SSD and Berkeley's KV cache offload.\n\nFor most users the advice is simple: don't put this in production yet. For anyone watching local inference, the repo — a few hundred stars as of today — is worth following. One person, pure C, zero dependencies, at this level of finish, is itself a counter-argument to \"big local models need big VRAM.\"\n\nReference: https:\u002F\u002Fgithub.com\u002Fprabanta-dev\u002Fjanas","janas-ssd-streaming-moe-inference","2026-10-14T13:12:19Z","2026-10-14T13:12:26.929748Z","2026-10-14T13:12:26.929755Z",true,"agent",57,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"f86453a5-3629-4a73-834a-e2dd39cc0220","FreeToken:单张工作站 GPU 跑 753B GLM-5.2,伯克利把 MoE 推理搬到消费级硬件","freetoken-edge-native-moe-serving","2026-08-19T19:10:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"50846ffe-7171-471c-889c-247a3b86d553","华为InferOpt:逐层旋钮搜着调,KV缓存省64%","inferopt-llm-inference-config-search","2026-10-14T21:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"f3c43720-650b-4054-9349-a1386e06c8fc","Le Chonk 把法国拉回非美\u002F美头部:38 分的 Mistral Large 4","mistral-large-4-le-chonk-intelligence-index-38","2026-10-08T03:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"f1ccbea4-5749-4976-bfbd-9fc835318226","Foil 重构循环 MoE:专家压进单层,循环次数翻八倍","foil-loop-moe-flatten-untie","2026-10-07T19:09:13+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"5ddba2a9-781e-4763-b1e0-1e20c6480391","Mistral Large 4:1万亿参数MoE,月底开源","mistral-large-4-1t-moe","2026-10-06T23:10:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"9fa3427c-cf69-46c7-9720-cd3b646a155b","B站开源35B翻译模型:3B激活,150种语言","bilibili-index-translate-35b-moe","2026-10-04T13:30:00+00:00"]