A 753B-parameter GLM-5.2 running on a single workstation GPU, a 284B model on a gaming desktop, a 35B model on an 8GB laptop GPU — this is not a marketing slide but the configuration list from FreeToken (arXiv:2608.16157), a paper released on August 17 by a UC Berkeley team. Within three days the paper climbed onto the Hugging Face Daily Papers trending list, and the author list includes Song Han, Matei Zaharia, and Ion Stoica.
Why Running Large MoE Models Locally Is Hard
Open-weight models keep getting larger — GLM-5.2 has 753B parameters, and DeepSeek and Qwen flagships are also MoE models in the hundreds of billions. Yet as the paper points out up front, serving systems for these models largely assume you have datacenter infrastructure.
Personal machines are a different reality. The paper's diagnosis: edge hardware exposes heterogeneous resources whose balance differs from machine to machine, and worse, agentic workloads continuously change their execution pattern — writing code one second, calling tools the next, with completely different expert activation distributions. Traditional offloading approaches are slow precisely because they treat a personal machine as a small GPU and patch around it.
FreeToken starts from a different premise: treat the whole machine — GPU, CPU, host memory, and interconnects — as one unified, elastic inference platform. Instead of committing to a fixed offloading strategy, it continuously maps computation and model state onto the resources actually available.
Three Engineering Cores
Bandwidth-adaptive CPU-GPU co-execution. The README calls it the q* policy: dynamically deciding which side executes expert computation based on bandwidth, combined with full-layer double-buffered prefill streaming and global LRU expert caching, so weight movement overlaps with compute.
Semantic-aware caching. Designed for agent workloads: context edits such as tool calls and thinking blocks reuse existing KV caches through semantic anchor checkpoints, avoiding redundant context recomputation. Anyone who has run long-session coding agents knows how painful that recomputation wait can be.
Elastic memory management. VRAM can be dynamically re-allocated between expert caches and KV memory at runtime — no engine restarts, no weight reloads. When the agent workload shifts, the memory layout shifts with it.
Reported Numbers and Ecosystem Position
Per the paper's reported figures: an 8GB laptop GPU can serve a 35B model, a gaming desktop can serve a 284B model, and a single workstation GPU can serve the 753B GLM-5.2. The system supports more than 20 open-weight MoE models — DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2 are named explicitly — with quantization formats covering MXFP4, NVFP4, FP8, and BF16, and native support for NVIDIA RTX 30/40/50 series consumer GPUs.
The interface layer deserves attention too: it exposes Anthropic/OpenAI-compatible APIs, and the README lists the coding agents it works with — Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness. In other words, your local Claude Code client can point its backend at a gaming PC running a 284B model.
The project is open-sourced under Apache 2.0, hosted on GitHub at FlashML-org/FreeToken, with desktop installers for Windows and Linux. The authors acknowledge the system was deeply inspired by mini-sglang and reuses code from SGLang, vLLM, and FlashInfer.
So What
What FreeToken really does is, in the paper's own words, turning open weights into deployable local software. Over the past two years open-weight models have closed the capability gap with closed ones, but the machines that could run them smoothly stayed in the datacenter; this paper pushes the boundary to devices users already own.
That said, all these numbers come from the authors' own report — real-world token throughput on consumer hardware still needs community validation. The project just open-sourced, GitHub stars are still modest, and the desktop app is at v0.2.0-beta. But the direction is clear: when a 753B model can live permanently on a single workstation GPU, the privacy, cost, and latency story of local agents gets rewritten all at once.