The teacher-is-oracle assumption in LLM KD is breaking\n\nFor the past two years, on-policy has been the default framing for LLM post-training: train on the model own generations, and you get sparser updates and better generalization. On the knowledge distillation (KD) track, however, there is an implicit assumption that has aged poorly — that the teacher is a reliable oracle. With modern LLMs, that assumption often fails: teacher predictions can carry high entropy, hallucination, and overconfident tails, and naively matching them with KL divergence can damage a well-calibrated student.\n\nAyan Sengupta, Vaibhav Seth, and Tanmoy Chakraborty (corresponding) at IIT Delhi posted Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models to arXiv (2609.36734) on September 29. The paper has been accepted at NeurIPS 2026. The team names the framework CaRE-KD, and the core idea is simple: stop trusting the teacher by default; use confidence as a gate.\n\n## Two-granularity design of CaRE-KD\n\nCaRE-KD operates at two granularities, one insurance policy per level.\n\nToken-level: CaRE-Divergence. Rather than committing to forward or reverse KL up front, CaRE-KD switches between them per token based on the relative confidence of teacher and student. When the teacher is more confident, the student is pushed to cover the teacher tails — closer to forward KL behavior. When the student is more confident, reverse KL pulls the student back, keeping it from chasing the teacher high-probability tails. The authors provide a gradient-level analysis showing that this conditional mechanism amounts to a conditional calibration that static divergences cannot reproduce.\n\nBatch-level: Revival. When a whole batch shows the teacher carrying more epistemic uncertainty than the student, Revival rejects the update outright. Revival is designed as a loss-agnostic plug-in — the paper shows it stacks on top of multiple existing KD objectives.\n\n## The numbers\n\nThe empirical sweep is broad: 8 teacher-student pairs, 11 benchmarks across instruction following, chat alignment, code generation, and mathematical reasoning. Average ROUGE-L improves +3.2 on instruction-following (over Skewed-KL, alpha-beta divergence), MBPP pass@1 +2.1, GSM8k +1.7, CollegeMath +1.8, and LLM-as-a-judge factuality up to +2.5 per task over Skewed-RKL. Robustness checks hold: removing gradient clipping, switching to sampled KL estimators, and training on longer-chain reasoning tasks all leave the core results intact.\n\n## Why this matters in 2026\n\nDistillation is now a strategic resource, not a training trick. Anthropic September report put 15.1 billion distillation-style requests from Alibaba, Moonshot, and DeepSeek on the table. CaRE-KD contribution is not just the lift; it is the explicit demolition of the teacher-is-reliable assumption, replaced by an engineering-ready confidence-gated framework backed by gradient-level theory. The bigger practical hook is that Revival is loss-agnostic and can be retrofitted into existing KD pipelines without rewriting the training loop.\n\n## My take\n\nDistillation has always been a trade-off between how reliable the teacher is and how much the student can learn. CaRE-KD answer is: stop pretending the teacher is perfect, and use confidence to dynamically calibrate trust. The same logic maps onto the wider LLM training landscape — RLHF, RLAIF, DPO all involve another-model-as-judge setups doing the same trust-and-calibrate dance. IIT Delhi team has turned that intuition into a working piece of training infrastructure. Worth a bookmark for anyone working on post-training stacks.