On June 7, 2026, the open-source organization Anemll put the Flash-iOS repository on GitHub — the first time a project of running a 400B-parameter-class MoE large model on an iPhone for local inference has taken on a "complete iOS port" form. It is based on developer Alexintosh's FlashMoE-iOS port, stacked with iOS memory entitlements, Fanout I/O, pread-only paths, and other low-level modifications, allowing what was originally only runnable on an M5 Max 128GB MacBook to have a "stuff into iPhone" possibility for the first time.
Three key technical points. First, the sparse activation of MoE is pushed to the extreme. Flash-iOS runs Qwen3.5-397B-A17B: 397B total parameters, but only 17B activated per token, 60 expert layers streamed by SSD on demand, pure C/Objective-C + handwritten Metal shader, held down by 6GB of memory. Second, Unsloth's 3-bit dynamic quantization is integrated into the MoE path for the first time: layer 27 attention stays in BF16, shared expert stays in BF16, routed expert goes to Q5_K, and other layers use mixed precision of IQ3_XXS/IQ4_XS, with 5.44 MB/expert size in exchange for near-4-bit PPL. Third, the iOS extended-virtual-addressing + increased-memory-limit and the pread-only design "around mmap" are the key engineering trade-offs for on-device ability to run large models.
But staying cool, the current 0.6 tok/s is far from practical — that means less than one Chinese character per second is generated. The real significance is: breaking the old constraint that "model weights must all reside in DRAM." As LLM parameter scale continues to push toward the trillion, the "SSD streaming inference + 3-bit mixed precision + MoE sparse activation" combination path may be the only realistic path to stuffing hundred-billion-parameter models into consumer devices. Apple Silicon's unified memory architecture makes this especially smooth, and for domestic open-source large models to replicate this path, the engineering chain's challenges are far greater than the model itself.