Recently accepted to arXiv and set to appear at ICCAD 2026, PolyQ offers a refreshingly new answer to the old problem of "low-bit LLM inference on edge devices": instead of obsessing over "uniformly compressing to 2/3/4-bit", it allocates bit widths from {2, 3, 4, 8, 16} across channels by importance under a fixed average-bit budget, then uses a paired compiler to reorganize heterogeneous-bit channels into SIMD/LUT-friendly homogeneous blocks — turning "fractional-bit deployment" from a concept into a practical CPU solution. Why does this matter? Current on-device LLM inference is stuck behind an invisible wall: GPUs following the NPU route are too expensive and power-hungry, while CPUs can't run dense models below 4-bit. PolyQ takes a more engineering-driven path — joint quantization-compiler design. Activation-aware per-channel bit allocation preserves accuracy; compile-time permutation and cross-operator fusion cut rearrangement traffic by 70.8% and push layout regularization entirely off the runtime path. On three very different model scales (Falcon-H1-3B, Llama2-13B, Qwen3-32B) and three CPU classes (Workstation / Laptop / Mobile), end-to-end validation shows: at a 3-bit target, perplexity improves 2.4–32.1% over prior work; prefill latency and decode throughput scale near-linearly with the bit budget; per-token energy cost increases by less than 2% relative to an optimized LUT backend. In other words, "3-bit on CPU" is not only viable but predictable in performance and energy. The engineering tradeoff is also worth noting: PolyQ doesn't chase "the lowest bit", but turns "fractional bits" into a tunable knob — echoing the recent idea used by DeepSeek, Kimi, and others with MoE + routing to "allocate compute on demand". The next competition in the LLM system stack will shift from "how hard can the model itself be compressed" to "how dry can the compiler-quantization collaboration squeeze the hardware". For mid-sized and small teams wanting to run local Agents, and for mobile/PC OEMs, this path is more worth betting on than simply scaling parameters.