Quantizing an agent's LLM to 4-bit cuts inference cost, but task success often quietly slides. SteerDistill (arXiv:2611.13410, submitted November 10, 2026) starts with a fine-grained diagnosis: a quantized agent does not get uniformly dumber. Instead, at one step of a multi-turn task it overlooks a task requirement or an environmental condition, and the environment state then diverges from the full-precision reference trajectory. In a multi-turn task, one wrong step is often enough to fail the whole task. The paper calls these drifts behavioral divergences, and fixing them is the entire point of the method.

Why the failures are hard to fix

The paper identifies two obstacles. Offline training data is unlikely to cover the exact contexts where the student model fails; letting the student roll out on its own rarely produces correct responses in those same contexts. The "correct demonstration" that classic distillation needs is absent from both sources.

Closed-loop teacher steering

SteerDistill comes from the Institute for AI Industry Research (AIR) at Tsinghua University, with six authors. The idea is to turn the full-precision model into an on-demand teacher. Teacher and student start from the same BF16 checkpoint, and the teacher stays frozen throughout. The student retries the divergent turn from the pre-error history and environment state; the teacher gives targeted guidance; whether a revision counts is judged by actually executing it — if it fails, the teacher refines the guidance using execution feedback until the correction passes local verification. Training proceeds in cycles: each cycle uses a frozen parameter snapshot in the low-bit runtime for rollout generation and revision, and the updated master weights are quantized into the next cycle's snapshot. During training, the student runs fake quantization in its forward pass and keeps BF16 master weights via a straight-through estimator.

Internalizing the corrections

You cannot ship a full-precision teacher alongside every deployment. SteerDistill trains the student without guidance: at corrected turns it matches the predictions of the steered student, and at other turns it matches the BF16 teacher. The corrections happen in the student's own failure contexts rather than in the distribution of external data; even when the whole task remains unsolved, locally verified corrections are still retained. At inference, the student runs alone — no teacher, no steering prompts.

The numbers (as reported by the paper)

  • Under W4A16 (4-bit weights, 16-bit activations), across three models and five agentic benchmarks, SteerDistill exceeds the strongest quantized baseline by 1.67–2.18 points on average and stays within one point of BF16.
  • On τ²-Bench, it beats QeRL by 1.84 percentage points in success rate while using 67% fewer training GPU hours, and its successful trajectories are shorter than the quantized baselines.
  • Under the more aggressive W2A16 (2-bit weights), it achieves the highest average score among the tested quantized baselines on Qwen3-8B.
  • Baselines include BitDistiller, On-Policy QAD, and QeRL: On-Policy QAD follows NVIDIA NeMo-RL's quantization-aware distillation workflow with top-64 teacher logits, while QeRL takes the GRPO reinforcement-learning route.

Two caveats

First, this is a preprint posted on November 10; every number above is self-reported by the paper, with no independent replication yet. Second, the code is on GitHub under an MIT license, but the checkpoints entry still reads TODO — readers who want to reproduce with the actual weights will have to wait. The README quickstart walks through Qwen3-8B on BFCL V4.

The paper turns "quantization hurts agents" from a blunt statistical observation into an engineering pipeline: locate the specific failing turn, fix it on the spot with a teacher in the loop, and distill the fix back into the parameters. For teams trying to squeeze agents onto edge devices, that path matters — the compute you save no longer has to be repaid in success rate. Paper: https://arxiv.org/abs/2611.13410 — Code: https://github.com/air-embodied-brain/SteerDistill