Ling-3.0-tiny: Ant Group's 1.3B-active KDA+MLA MoE running end-to-end on M4 Pro — FP8 8.3 GiB peak, INT4 115 tok/s

Background

On August 17, 2026, Ant Group's InclusionAI released Ling-3.0-tiny on Hugging Face — a 7.9B-total / ~1.2–1.3B-active hybrid hybrid linear-attention Mixture-of-Experts model. It is the smallest member of the Ling-3.0 family, targeting production-grade Agent workloads on Apple Silicon MacBook and NVIDIA DGX Spark. Three precision checkpoints (BF16, FP8, INT4) ship simultaneously under the MIT license, with same-day support from SGLang (cookbook), vLLM, and Ollama (community PR #17643).

This is not just another small model. It ports the KDA (linear attention) + MLA (multi-head latent attention) hybrid stack — previously validated only in frontier-scale models — end-to-end down to ~1.2–1.3B active parameters.

Core architecture: 5:1 vs 3:1

Ling-3.0-tiny sits next to the earlier Ling-3.0 Flash (124B/5.1B) but uses a different attention ratio:

  • Ling-3.0 Flash: 35 KDA + 7 Gated MLA layers, 5:1 ratio, 512 routed experts (8 active)
  • Ling-3.0-tiny: 3 KDA + 1 MLA layers per 4-layer block, 3:1 ratio, 128 routed experts (8 active + 1 shared)

Ollama PR #17643's implementation notes describe the engineering: each 4-layer block alternates 3 KDA linear-recurrent layers with 1 MLA layer, sitting on top of a sparse MoE FFN. KDA uses short convolutions + segmented delta-rule scan + safe-gate decay + sigmoid-gated RMSNorm; MLA uses 576-dim compressed latent + interleaved RoPE + an absorbed latent KV cache — caching the 576-dim compressed latent directly instead of expanding to per-head K/V, cutting KV memory by ~10×.

Because active parameters are only ~1.2–1.3B, every skipped full-attention layer saves KV cache memory amplification — that tradeoff is more sensitive on edge devices, so tiny raises KDA density and MLA only takes 25%. Flash inverts this: at frontier scale, KV cache is a small fraction of total memory, so it can safely give MLA more density.

On-device numbers

InclusionAI validated three local hardware targets:

Hardware Precision Output speed Peak memory
Apple M4 Pro MacBook (48GB unified) FP8 ~105 tok/s (after Ollama MLX optimization) ~8.3 GiB (8K context)
Apple M4 Pro MacBook INT4 ~115 tok/s ~6.1 GiB
Apple M4 Pro MacBook BF16 ~68–71 tok/s ~14.9 GiB
NVIDIA DGX Spark FP8 100–105 tok/s —
Mac mini (MLX backend, Ollama PR #17643) BF16/FP8/INT4 same edge target —

The Ollama team also ran long-context stress tests: Ling-3.0 Flash FP4 at 128K prefill completes 122,292-token full prefill on GB10 (128GB) at ~96.7 GiB peak; tiny, on the same MLA chunked prefill path, hits 429 tok/s prefill, 89 tok/s decode for 8K prompts on M4 Pro, peaking at 28.3 GiB.

Artificial Analysis's end-to-end test: Ling-3.0-tiny completes a 500-token response (including thinking) in 18 seconds, sustains 160+ tok/s steady state, scores 25 on the AA Intelligence Index v4.1.1 and 16 on the AA Agentic Index. SGLang's own GSM8K measurement: 94.01% (single GPU BF16).

BF16 / FP8 / INT4: same weights, three hardware paths

Three precision checkpoints ship simultaneously:

  • BF16: ~15.8 GB; default in SGLang and vLLM
  • FP8 (inclusionAI/Ling-3.0-tiny-fp8): blockwise E4M3 (128×128) with dynamic activation; attention projections, dense MoE gate, lm_head kept at higher precision; ~7.9 GB
  • INT4 (inclusionAI/Ling-3.0-tiny-int4): compressed-tensors W4A16 routed experts; ~5.8 GB

MLX quantization accuracy validation (Ollama PR measurements, CUDA torch bf16 reference):

Precision rel L2 cosine argmax peak memory (M4 Pro) decode (M4 Pro)
BF16 0.027 0.9996 match ~14.9 GiB 68–71 tok/s
FP8 → MXFP8 0.077 0.9970 match ~8.3 GiB ~105 tok/s
INT4 0.153 0.9883 match ~6.1 GiB ~115 tok/s

Note INT4 runs faster than FP8 — because INT4 routed experts use GatherQMM end-to-end on the quantized path, never materializing BF16 experts, saving dequant + recompute memory bandwidth and kernel-launch overhead.

The official Model Card ships deploy scripts directly: docker pull lmsysorg/sglang:dev-Ling-3.0-tiny for SGLang's prebuilt image, with 256K YaRN context activated via --json-model-override-args (ROPE scaling + partial_rotary_factor=0.5). The full recipe fits on a single H20-3e (141GB) or a single Blackwell node.

10,000+ Agentic training environments

Like Ling-3.0 Flash, tiny was trained end-to-end across 10,000+ interactive environments covering Coding, General, and Deep Research Agent tasks. The native SGLang HiCache + Mooncake hierarchical cache (physical dual-pool + cluster L3 cache) eliminates redundant recomputation during long-horizon interactions, reducing TTFT by 60–80% on long-input scenarios.

Thinking is on by default; toggle off per request with chat_template_kwargs.enable_thinking=false. SGLang defaults to --reasoning-parser deepseek-r1 and --tool-call-parser glm45 (different from Flash's ling3, because the Bailing Chat Template uses <role> + <arg_key>/<arg_value> structure).

Why this release matters

In the August 2026 landscape:

  1. Edge LLMs reach "thinking-on + long-context-no-degradation". Apple Silicon hitting 105 tok/s FP8 + 256K prefill at 429 tok/s + 6.1 GiB INT4 means an entry-level 16GB MacBook can run INT4 weights with thinking mode without throttling. LFM2.5-2.6B (220 tok/s, 2.6B dense), Qwen3.6-35B-A3B, and prior edge-class models all reached 200+ tok/s — but thinking-on-by-default + 256K context without slowdown + KDA linear attention down to ~1.3B active params is new.

  2. KDA-class linear attention leaves arXiv. It was still arXiv-only in late 2025; in H1 2026 it got validated in frontier models (Kimi K3, Kimi Linear). Ling-3.0-tiny ports KDA + MLA end-to-end to the 1.2–1.3B-active range — "linear attention + MoE" as a complete recipe on edge. The Ollama PR needed nearly 10 commits (absorbed-MLA cache, KDA chunked prefill scan, etc.) to stabilize the MLX implementation — the engineering bar is high.

  3. Chinese open-source ecosystem fills the mid-size gap for the Agent era. Qwen3.6 and Kimi K3 are flagships (>100B); Ling-3.0-tiny, LFM2.5-2.6B, LFM2.5-VL-3B are edge (1–8B). The 8B–30B middle is still contested, but edge production-ready models are now shipping fast enough that Agent frameworks (Hermes Agent, OpenClaw, Cline, OpenHands) can route through local models instead of cloud APIs.

What I'll be watching

  • Independent Agent workload reproductions: SWE-Bench Pro / MCP-Atlas / BrowseComp, not just AA Intelligence Index or GSM8K. The official evaluation covers them but doesn't publish independent reproduction numbers.
  • Whether 1-bit quantization can survive on MoE + KDA. Kimi K3's MXFP4 + QAT is a precedent, but the 1.3B-active subnetwork under more aggressive quantization is the key question for true edge ubiquity.
  • Apple Silicon NPU / Neural Engine acceleration. Current Ollama + MLX runs the GPU path; ANE support for Ling-3.0-tiny would be the next big edge-LLM event.

References