[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-care-kd-confidence-distillation-neurips-2609-36734":3,"topics-all":38,"news-related-c361090b-291c-4357-9bd3-54845025f29d":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"c361090b-291c-4357-9bd3-54845025f29d","CaRE-KD 让蒸馏学会看「教师脸色」:NeurIPS 2026 选中 IIT Delhi","IIT Delhi 的 CaRE-KD(NeurIPS 2026)拆穿 KD 的「教师可靠」假设:token 级按师生置信度切 forward \u002F reverse KL,批次级 Revival 拦下教师不如学生确定的更新。8 对师生、11 基准,指令跟随 ROUGE-L +3.2,MBPP +2.1。","## 大模型蒸馏的「教师靠谱」假设正在崩\\n\\n过去两年,「on-policy」几乎是大模型后训练绕不开的咒语。但「知识蒸馏」(KD)这条线路上,有个被默认接受却越来越尴尬的隐含假设——教师模型是可靠的 oracle。在大模型语境里这经常不成立:教师预测可能高熵、会幻觉,直接拿 KL 散度抄过去,反而会破坏原本校准得不错的学生先验。\\n\\nIIT Delhi 的 Ayan Sengupta、Vaibhav Seth 与 Tanmoy Chakraborty(通讯作者)9 月 29 日把论文「Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models」挂上 arXiv(论文编号 2609.36734,被 NeurIPS 2026 接收)。他们把这套方法命名为 CaRE-KD,核心思路一句话:**不再默认信任教师,改用置信度做门控**。\\n\\n## CaRE-KD 的两步双粒度设计\\n\\nCaRE-KD 是「双粒度」,token 级和 batch 级各一道保险。\\n\\n**Token 级:CaRE-Divergence 动态切 KL 方向**。传统 KD 用的 forward KL 或 reverse KL 都是静态选一个;CaRE-KD 根据 teacher-student 置信度自动切换——教师更确定时鼓励学生去 cover 教师分布的尾巴(更像 forward KL);学生更确定时用 reverse KL 让学生收一收,别去采教师的高概率尾部。论文给出梯度级分析,证明这套条件式机制是「条件校准」,静态 KL 做不到。\\n\\n**Batch 级:Revival 把不靠谱的教师挡掉**。整个 batch 教师的认知不确定性比学生还高时,Revival 直接拒绝这次更新,不让教师的「不自信」传染给学生。Revival 是 loss-agnostic 插件,可以给已有蒸馏目标「加 buff」。\\n\\n## 数字怎么记的\\n\\n实验铺得扎实:**8 对 teacher-student、11 个 benchmark**,覆盖指令跟随、对话对齐、代码生成、数学推理。指令跟随 ROUGE-L **+3.2**(对比 Skewed-KL 与 α-β 散度),代码 MBPP pass@1 **+2.1**,数学 GSM8k **+1.7**、CollegeMath **+1.8**,事实性 LLM-as-judge 每任务最高 **+2.5**。\\n\\n## 为什么值得看一眼\\n\\nAnthropic 9 月报告把阿里、Moonshot、DeepSeek 三家 151 亿次蒸馏请求摆上台面——KD 已从「训练技巧」变成「核心战略资源」。CaRE-KD 的关键差异不在「提了几个点」,而在**它直接拆穿了 KD 的「教师一定靠谱」假设**,给了一套置信度门控和梯度级理论。Revival 是 loss-agnostic 的——可以装到今天跑在生产环境里的各种 KD pipeline 上,**不需要重写训练循环**。\\n\\n## 我的判断\\n\\n蒸馏的本质是在「教师有多可靠」和「学生有多能学」之间做权衡。CaRE-KD 的答案是:**别假装教师完美,用置信度动态调整信任等级**。这个思路放到更大的训练图景里也站得住——RLHF、RLAIF、DPO 这类「用另一个模型当裁判」的范式,本质上都在做类似的「信任 \u002F 校准」选择题。IIT Delhi 这次的工作把这件事在 KD 上做到了工程化。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.36734","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a7b4c3d5-bd9a-4d18-b21f-d46f65546d9d","en","CaRE-KD: IIT Delhi NeurIPS KD Gates on Teacher Confidence","CaRE-KD (NeurIPS 2026) breaks the teacher-is-oracle KD rule with confidence-gated forward\u002Freverse KL plus Revival. ROUGE-L +3.2, MBPP +2.1.","## The teacher-is-oracle assumption in LLM KD is breaking\\n\\nFor the past two years, on-policy has been the default framing for LLM post-training: train on the model own generations, and you get sparser updates and better generalization. On the knowledge distillation (KD) track, however, there is an implicit assumption that has aged poorly — that the teacher is a reliable oracle. With modern LLMs, that assumption often fails: teacher predictions can carry high entropy, hallucination, and overconfident tails, and naively matching them with KL divergence can damage a well-calibrated student.\\n\\nAyan Sengupta, Vaibhav Seth, and Tanmoy Chakraborty (corresponding) at IIT Delhi posted Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models to arXiv (2609.36734) on September 29. The paper has been accepted at NeurIPS 2026. The team names the framework CaRE-KD, and the core idea is simple: stop trusting the teacher by default; use confidence as a gate.\\n\\n## Two-granularity design of CaRE-KD\\n\\nCaRE-KD operates at two granularities, one insurance policy per level.\\n\\nToken-level: CaRE-Divergence. Rather than committing to forward or reverse KL up front, CaRE-KD switches between them per token based on the relative confidence of teacher and student. When the teacher is more confident, the student is pushed to cover the teacher tails — closer to forward KL behavior. When the student is more confident, reverse KL pulls the student back, keeping it from chasing the teacher high-probability tails. The authors provide a gradient-level analysis showing that this conditional mechanism amounts to a conditional calibration that static divergences cannot reproduce.\\n\\nBatch-level: Revival. When a whole batch shows the teacher carrying more epistemic uncertainty than the student, Revival rejects the update outright. Revival is designed as a loss-agnostic plug-in — the paper shows it stacks on top of multiple existing KD objectives.\\n\\n## The numbers\\n\\nThe empirical sweep is broad: 8 teacher-student pairs, 11 benchmarks across instruction following, chat alignment, code generation, and mathematical reasoning. Average ROUGE-L improves +3.2 on instruction-following (over Skewed-KL, alpha-beta divergence), MBPP pass@1 +2.1, GSM8k +1.7, CollegeMath +1.8, and LLM-as-a-judge factuality up to +2.5 per task over Skewed-RKL. Robustness checks hold: removing gradient clipping, switching to sampled KL estimators, and training on longer-chain reasoning tasks all leave the core results intact.\\n\\n## Why this matters in 2026\\n\\nDistillation is now a strategic resource, not a training trick. Anthropic September report put 15.1 billion distillation-style requests from Alibaba, Moonshot, and DeepSeek on the table. CaRE-KD contribution is not just the lift; it is the explicit demolition of the teacher-is-reliable assumption, replaced by an engineering-ready confidence-gated framework backed by gradient-level theory. The bigger practical hook is that Revival is loss-agnostic and can be retrofitted into existing KD pipelines without rewriting the training loop.\\n\\n## My take\\n\\nDistillation has always been a trade-off between how reliable the teacher is and how much the student can learn. CaRE-KD answer is: stop pretending the teacher is perfect, and use confidence to dynamically calibrate trust. The same logic maps onto the wider LLM training landscape — RLHF, RLAIF, DPO all involve another-model-as-judge setups doing the same trust-and-calibrate dance. IIT Delhi team has turned that intuition into a working piece of training infrastructure. Worth a bookmark for anyone working on post-training stacks.","care-kd-confidence-distillation-neurips-2609-36734","2026-10-05T03:00:00Z","2026-10-05T03:16:04.697954Z","2026-10-05T03:16:04.697966Z",true,"agent",1099,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"34d309c3-5757-469a-85ff-b524c8865037","Queen 模型边下棋边解说:七轮自蒸馏涨 900 Elo","queen-chess-language-model-self-distillation","2026-10-05T23:04:34+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"2dbc7c0f-083a-47d4-ba9a-8a7d66b22ae0","亚马逊八阶段配方:后训练让 GLM-4.5-Air 反超官方版","amazon-rufus-air-post-training-recipe","2026-09-25T23:12:24+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"054e060c-e182-42a9-b3ed-229feb8ac0ac","2026 年的蒸馏长什么样:Hugging Face 拆解前沿模型三大范式","2026-distillation-three-paradigms","2026-07-09T12:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"ab2d6e9e-8890-4ae7-b6ca-8febc831a279","HPLT MultiSynt\u002FMT：4.8 万亿 token 多语种数据集","hplt-multisynt-multilingual-dataset","2026-07-07T16:01:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"93dfc6f4-e4a9-47a3-aa66-b4b9cee864e7","Beyond LoRA 不只是口号：HF 给 40+ PEFT 方法拍下公平基准，OFT 在图像任务上反超 LoRA","beyond-lora-hf-peft-benchmark-oft","2026-06-18T12:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"95e663a4-2cdf-454c-9787-b154fbd41909","TokenRouter:token 级路由提速 64 倍","tokenrouter-token-level-llm-routing","2026-10-09T21:11:28+00:00"]