[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-freetoken-edge-native-moe-serving":3,"news-related-f86453a5-3629-4a73-834a-e2dd39cc0220":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"f86453a5-3629-4a73-834a-e2dd39cc0220","FreeToken:单张工作站 GPU 跑 753B GLM-5.2,伯克利把 MoE 推理搬到消费级硬件","UC Berkeley 团队发布边缘原生 MoE 推理引擎 FreeToken:通过带宽自适应的 CPU-GPU 协同执行与弹性内存管理,让 8GB 笔记本 GPU 跑 35B 模型、单张工作站 GPU 跑 753B GLM-5.2,支持 20 多个开源 MoE 模型,已按 Apache 2.0 开源。","753B 参数的 GLM-5.2 跑在单张工作站 GPU 上,284B 模型塞进游戏主机,35B 模型装进 8GB 显存的笔记本——这不是营销幻灯片,而是 UC Berkeley 团队 8 月 17 日放出的论文 FreeToken(arXiv:2608.16157)给出的配置清单。论文上线三天,已登上 Hugging Face Daily Papers 热榜,作者名单里包括 Song Han、Matei Zaharia 和 Ion Stoica。\n\n## 本地跑大 MoE,到底难在哪\n\n开源权重的模型越做越大——GLM-5.2 有 753B 参数,DeepSeek、Qwen 的旗舰也是数百 B 级别的 MoE。但论文开篇就点破现状:为这些模型做 serving 的系统,基本都默认你有一台数据中心服务器。\n\n个人机器的现实完全不同。论文的判断是:边缘硬件暴露的是异构资源,而且每台机器的配比都不一样;更麻烦的是,agent 工作流的执行模式还在持续变化——上一秒在写代码,下一秒在调工具,专家激活的分布完全不同。传统的 offload 方案之所以慢,是因为它们把个人机器当成\"一块小 GPU\"来打补丁。\n\nFreeToken 的出发点是把整台机器——GPU、CPU、host 内存、互连——当成一个统一的弹性推理平台来调度。它不固定任何 offloading 策略,而是持续地把计算和模型状态映射到实际可用的资源上。\n\n## 三个工程核心\n\n**带宽自适应的 CPU-GPU 协同执行。** 项目 README 称之为 q* 策略:根据带宽动态决定专家计算放在哪一侧执行,配合全层 double-buffered prefill 流式加载和全局 LRU 专家缓存,尽量让权重搬运和计算重叠。\n\n**语义感知缓存。** 这是专门为 agent 场景设计的:工具调用、思考块这类上下文编辑,通过 semantic anchor 检查点复用已有的 KV cache,避免整段上下文重算。跑过长会话 coding agent 的人都知道,上下文重算的等待有多折磨人。\n\n**弹性内存管理。** 显存里的专家缓存和 KV 内存可以在运行时动态再分配,不用重启引擎、不用重载权重——agent 负载变了,内存布局跟着变。\n\n## 实测口径与生态位\n\n按论文报告的数字:8GB 显存的笔记本可以跑 35B 模型,一台游戏桌面可以跑 284B 模型,单张工作站 GPU 可以跑 753B 的 GLM-5.2。系统支持 20 多个开源 MoE 模型,官方点名 DeepSeek-V4-Flash、Qwen3.6-35B-A3B 和 GLM-5.2,量化格式覆盖 MXFP4、NVFP4、FP8、BF16,硬件支持 NVIDIA RTX 30\u002F40\u002F50 系列消费卡。\n\n更值得注意的是接口层:它提供 Anthropic\u002FOpenAI 兼容 API,README 直接列出了适配的 coding agent——Codex、Claude Code、OpenCode、OpenClaw、DeepSeek Harness。也就是说,你本地的 Claude Code 完全可以把后端指向这台跑着 284B 模型的游戏 PC。\n\n项目以 Apache 2.0 许可证开源,代码托管在 GitHub 的 FlashML-org\u002FFreeToken,桌面端提供 Windows 和 Linux 安装包。作者也坦言系统深受 mini-sglang 启发,复用了 SGLang、vLLM、FlashInfer 等项目的代码。\n\n## 所以呢\n\nFreeToken 真正做的事,是论文原话说的 \"turns open weights into deployable local software\"——把开源权重变成可部署的本地软件。过去两年开源模型在能力上追平闭源,但能流畅跑起来的机器始终在数据中心;这篇论文把边界推到了用户手头已有的设备上。\n\n当然,上面这些数字都来自作者自己的报告,消费级硬件上的实际 token 速度还需要社区验证——项目刚开源,GitHub star 数还不多,桌面端也还处于 v0.2.0-beta 阶段。但方向已经很清楚:当 753B 模型可以常驻一张工作站 GPU,本地 agent 的隐私、成本和延迟三件事,会被同时改写。\n\n论文地址:https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.16157","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.16157","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"53dc8c6e-70bd-499a-ab2f-222504b375b0","en","FreeToken: UC Berkeley Brings 753B GLM-5.2 MoE Inference to Consumer Hardware","UC Berkeley researchers released FreeToken, an edge-native MoE serving engine with bandwidth-adaptive CPU-GPU co-execution and elastic memory management. It runs a 35B model on an 8GB laptop GPU and the 753B GLM-5.2 on a single workstation GPU, supports 20+ open-weight MoE models, and is open-sourced under Apache 2.0.","A 753B-parameter GLM-5.2 running on a single workstation GPU, a 284B model on a gaming desktop, a 35B model on an 8GB laptop GPU — this is not a marketing slide but the configuration list from FreeToken (arXiv:2608.16157), a paper released on August 17 by a UC Berkeley team. Within three days the paper climbed onto the Hugging Face Daily Papers trending list, and the author list includes Song Han, Matei Zaharia, and Ion Stoica.\n\n## Why Running Large MoE Models Locally Is Hard\n\nOpen-weight models keep getting larger — GLM-5.2 has 753B parameters, and DeepSeek and Qwen flagships are also MoE models in the hundreds of billions. Yet as the paper points out up front, serving systems for these models largely assume you have datacenter infrastructure.\n\nPersonal machines are a different reality. The paper's diagnosis: edge hardware exposes heterogeneous resources whose balance differs from machine to machine, and worse, agentic workloads continuously change their execution pattern — writing code one second, calling tools the next, with completely different expert activation distributions. Traditional offloading approaches are slow precisely because they treat a personal machine as a small GPU and patch around it.\n\nFreeToken starts from a different premise: treat the whole machine — GPU, CPU, host memory, and interconnects — as one unified, elastic inference platform. Instead of committing to a fixed offloading strategy, it continuously maps computation and model state onto the resources actually available.\n\n## Three Engineering Cores\n\n**Bandwidth-adaptive CPU-GPU co-execution.** The README calls it the q* policy: dynamically deciding which side executes expert computation based on bandwidth, combined with full-layer double-buffered prefill streaming and global LRU expert caching, so weight movement overlaps with compute.\n\n**Semantic-aware caching.** Designed for agent workloads: context edits such as tool calls and thinking blocks reuse existing KV caches through semantic anchor checkpoints, avoiding redundant context recomputation. Anyone who has run long-session coding agents knows how painful that recomputation wait can be.\n\n**Elastic memory management.** VRAM can be dynamically re-allocated between expert caches and KV memory at runtime — no engine restarts, no weight reloads. When the agent workload shifts, the memory layout shifts with it.\n\n## Reported Numbers and Ecosystem Position\n\nPer the paper's reported figures: an 8GB laptop GPU can serve a 35B model, a gaming desktop can serve a 284B model, and a single workstation GPU can serve the 753B GLM-5.2. The system supports more than 20 open-weight MoE models — DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2 are named explicitly — with quantization formats covering MXFP4, NVFP4, FP8, and BF16, and native support for NVIDIA RTX 30\u002F40\u002F50 series consumer GPUs.\n\nThe interface layer deserves attention too: it exposes Anthropic\u002FOpenAI-compatible APIs, and the README lists the coding agents it works with — Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness. In other words, your local Claude Code client can point its backend at a gaming PC running a 284B model.\n\nThe project is open-sourced under Apache 2.0, hosted on GitHub at FlashML-org\u002FFreeToken, with desktop installers for Windows and Linux. The authors acknowledge the system was deeply inspired by mini-sglang and reuses code from SGLang, vLLM, and FlashInfer.\n\n## So What\n\nWhat FreeToken really does is, in the paper's own words, turning open weights into deployable local software. Over the past two years open-weight models have closed the capability gap with closed ones, but the machines that could run them smoothly stayed in the datacenter; this paper pushes the boundary to devices users already own.\n\nThat said, all these numbers come from the authors' own report — real-world token throughput on consumer hardware still needs community validation. The project just open-sourced, GitHub stars are still modest, and the desktop app is at v0.2.0-beta. But the direction is clear: when a 753B model can live permanently on a single workstation GPU, the privacy, cost, and latency story of local agents gets rewritten all at once.\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.16157","freetoken-edge-native-moe-serving","2026-08-19T19:10:00Z","2026-08-19T19:08:11.092544Z","2026-08-19T19:08:11.092553Z",true,"agent",796,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"804ab59a-a8d6-4b61-bf74-8f6f2bdae83c","智谱把 Flash 做成一件正经事:一次说清 GLM-5.3-Flash 的架构和 benchmark 真相","glm-5-3-flash-hybrid-attention-architecture","2026-08-27T08:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"68072ee1-fc37-4064-ab18-09550ae72d1b","GLM-5.3-Flash 把 320B MoE 跑在国产芯片上:Flash 价位和 $0.15 API 的混合注意力栈","glm-5-3-flash-chinese-chips-hybrid-attention","2026-08-27T03:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"cac485ab-a429-4ebc-88c6-a1f924f978ff","AWS 开源 KeysAndValues:微调时就让模型学会“遗忘”,单张 A100 撑住 128K","aws-keysvalues-sparse-attention-finetuning","2026-08-26T05:20:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"c94bdf86-5de9-49fe-8c98-0f5c47611bfe","SGLang v0.5.18 发布:大模型冷启动提速 2.38 倍,710 个 PR 都改了什么","sglang-v0-5-18-cold-start-2-38x","2026-08-24T23:15:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"1311adb6-dc19-41a7-a188-6760d9e53672","HF Summer 2026 报告:13 个下载量 Top 25 模型是 2022 年的老面孔","hugging-face-summer-2026-attention-adoption","2026-08-24T08:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"988bbfb8-672a-4c6c-98f7-3a170b6bd8b3","Macaw 把 LFM2.5 装进 1.5GB:4-bit 端侧 LLM 跑 Mac 控制工具链","macaw-lfm25-15gb-edge-mac-agent","2026-08-24T06:00:00+00:00"]