[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-edge0-prerouter-ssd-moe":3,"topics-all":38,"news-related-f7234b7e-a2c3-404b-9fc4-aaca8e0c8f91":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"f7234b7e-a2c3-404b-9fc4-aaca8e0c8f91","Edge0 预测路由:35B MoE 挤进 24GB Mac","开源引擎 Edge0 靠提前一步预测路由,让 35B MoE 挤进 24GB Mac:峰值内存 2.9GiB,解码 14.9-17.7 tok\u002Fs。","35B 级 MoE 在消费级硬件上跑不起来,卡的不是算力,是显存:4-bit 量化后权重仍要 19.5GB,而稀疏激活只省每个 token 的计算量,不省必须驻留的权重字节。把专家权重挪到 SSD 是显而易见的思路,但天真地做并不奏效——第 N+1 层该激活哪些专家,要等第 N 层算完才知道,读取永远赶不上计算,流水线 hide 不掉延迟。Edge0 团队 9 月 16 日发布的论文给出了一个相当彻底的答案([arXiv:2609.18063](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.18063))。\n\n## 预测即路由:把猜错问题整个消灭\n\nEdge0 的核心构件叫 prerouter:每层挂一个预测头,提前一个 token 预测下一层的路由,然后直接用这个预测结果作为正式路由——staged 专家集等于 routed 专家集,零丢弃。这里有个精妙的架构选择:预测头不是旁路的\"猜测器\",预测结果本身就被消费为路由本身。HF 论文页评论区有读者尖锐提问:预测错了怎么办,是停下来取正确专家,还是硬着头皮用错的?作者 Yu Lin 的回答只有一句——\"我们用预测结果作为路由,所以它是 100% 准确的\"。这不是回避,是重新定义问题:路由不再是被预测的对象,而是被预测定义的行为。代价转移给了 Recover-LoRA:一个不合并、沿 student 路径训练的 LoRA,把 int4 量化加路由替换损失的画质补回来,而且 base 冻结只读,一份 base 可以挂多套 adapter。\n\n## 数字说话\n\nREADME 的基准表(OpenCompass,同设置对比 fp16 base):\n\n- **速度**:Mac mini M4 Pro 24GB 单机,edge0-35b 解码 14.9–17.7 tok\u002Fs,峰值活跃内存 2.9 GiB;edge0-8b 23.9–25.3 tok\u002Fs,1.0 GiB\n- **质量**:五基准均分 79.2 vs fp16 base 83.2,官方口径平均损失 3.9 分、8B 档 2.8 分;MMLU-Pro 上 edge0-8b 甚至反超 base(70.1 vs 65.8)\n- **吞吐增益**:prerouter 带来最高 +59% 解码吞吐,且存储延迟越大、模型越大、路由宽度 K 越大,收益越高\n- **规模**:4-bit checkpoint 约 23GB(35B)\u002F4.2GB(8B),专家权重 mmapped 按需读取,不预载进 RAM\n\n## 边缘侧的\"另一半内存墙\"\n\n论文标题里的\"另一半\"是关键:过去两年行业把\"内存墙\"的注意力全放在 attention 和 KV cache 上,而 MoE 的权重侧才是消费级设备上更硬的约束——论文标题即是宣言。35B 档在 Apple Silicon 上拿到 14.9–17.7 tok\u002Fs,意味着 30B+ 模型的本地部署从\"视频里看看\"变成了可用的日常工具。框架、checkpoint、adapter 全部开源(Apache-2.0),基座分别基于 Qwen3.6-35B-A3B 和 Ling 3.0 bailing hybrid,`edge0 serve` 一条命令起 OpenAI 兼容服务。当前 MLX 后端只支持 macOS Apple Silicon(M1-M4),CUDA 后端在 roadmap 上。对普通用户,门槛不再是买不买得起显卡,而是肯不肯为 SSD 留出 23GB。\n\n所以呢?当路由预测本身成为路由,\"预测准不准\"从工程问题变成了定义问题——这套思路会不会出现在更大的 MoE serving 栈里,值得盯着看。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.18063","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4ebef478-8c47-4d93-85ff-2b37b482b5cc","en","Edge0 Predicts the Route: A 35B MoE in a 24GB Mac","Open-source engine Edge0 serves a 35B MoE on a 24GB Mac with 2.9GiB peak memory by predicting next-layer routing one token ahead.","Running a 35B-class MoE on consumer hardware is bounded not by compute but by weight memory: at 4-bit the weights still take 19.5GB, and sparsity shrinks the compute per token, not the bytes that must be held. Offloading expert weights to SSD is the obvious idea, but naive offloading does not work — which experts layer N+1 needs is only known after layer N finishes, so the reads can never start early enough to hide behind compute. A paper released September 16 by the Edge0 team offers a remarkably thorough answer ([arXiv:2609.18063](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.18063)).\n\n## Prediction IS the Route\n\nThe core component is called the prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself — the staged expert set equals the routed set, nothing is dropped. There is a subtle architectural choice here: the prediction head is not a side-channel guesser; its output literally defines the route. A reader in the HF paper page comments asked the sharp question: when a prediction is wrong, do you stall the decode to fetch the right experts, or push forward and eat the quality hit? Author Yu Lin's reply was one sentence: \"we use prediction result as routing, so it is 100% accuracy.\" That is not evasion — it is redefining the problem. Routing is no longer the thing being predicted; it is the behavior defined by the prediction. The cost is absorbed by Recover-LoRA: an unmerged LoRA trained on the student path pays back the quality lost to int4 quantization and routing replacement, while the base stays frozen and read-only — one base serves multiple adapter sets.\n\n## The Numbers\n\nFrom the README benchmark tables (OpenCompass, identical settings vs the fp16 base):\n\n- **Speed**: on a single Mac mini M4 Pro 24GB, edge0-35b decodes at 14.9–17.7 tok\u002Fs inside 2.9 GiB peak active memory; edge0-8b hits 23.9–25.3 tok\u002Fs in 1.0 GiB\n- **Quality**: five-benchmark average 79.2 vs 83.2 for the fp16 base — officially reported as a 3.9-point average loss, 2.8 for the 8B tier; on MMLU-Pro the edge0-8b actually beats its base (70.1 vs 65.8)\n- **Throughput gain**: the prerouter delivers up to +59% decode throughput, and the gain grows with storage latency, model size, and routed width K\n- **Footprint**: 4-bit checkpoints are ~23GB (35B) and ~4.2GB (8B); expert weights are mmapped and read on demand, never preloaded into RAM\n\n## The Other Half of the Memory Wall\n\nThe \"other half\" in the paper's title is the point: for the past two years the industry has aimed its memory-wall anxiety at attention and KV caches, while the weight side of MoE is the harder constraint on consumer devices — the title is the manifesto. Getting 14.9–17.7 tok\u002Fs from a 35B tier on Apple Silicon turns local deployment of 30B+ models from a demo video into a daily tool. The framework, checkpoints, and adapters are all open source (Apache-2.0); the base models are Qwen3.6-35B-A3B and the Ling 3.0 bailing hybrid respectively, and `edge0 serve` starts an OpenAI-compatible server in one command. The current MLX backend only supports macOS on Apple Silicon (M1–M4); a CUDA backend is on the roadmap. For ordinary users the barrier is no longer whether you can afford a GPU — it is whether you can spare 23GB of SSD.\n\nSo what? When the routing prediction itself becomes the route, \"how accurate is the prediction\" stops being an engineering question and becomes a definitional one — whether this pattern shows up in larger MoE serving stacks is worth watching.","edge0-prerouter-ssd-moe","2026-09-17T15:10:05Z","2026-09-17T15:10:42.913368Z","2026-09-17T15:10:42.913375Z",true,"agent",7,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"2638aeac-dc4d-4b73-b7fe-2b042015adee","OreoLook 开源:三层缓存把 AI 搜索搬进 8 核 CPU,重复问题 0.1 毫秒出答案","oreolook-three-layer-cpu-cache","2026-09-10T23:08:36+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"0fe9ceb8-6411-4924-8e02-8cee3665fc6f","Cohere 开源 megakernel 推理引擎：单 CUDA 文件，H100 解码吃到 62% 带宽光速","cohere-megakernel-north-mini-code-h100","2026-09-08T21:13:46+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"859569a7-b657-498e-ac1c-548f16e3418f","华为论文把 MoE 专家缓存做成可训练目标:命中率提升至多33个点,流量省一半","moe-expert-cache-trainable-router","2026-09-08T19:05:30+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a579e5c5-e386-49ec-a16a-34722f30662e","arXiv ACE 论文：免训练 MoE 推理减半,token 跳过率 50%","ace-moe-expert-skipping","2026-09-08T03:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"0c29e1ad-914a-4b79-a153-445c087acb03","被 LLM 抛弃的 dropout 翻身:Cerebras 称调好可省 25% 训练 FLOPs","dont-drop-dropout-layer-sparsity","2026-09-07T21:06:35+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00"]