[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-ambient-sft-temperature-matching":3,"topics-all":38,"news-related-8e741f04-82a2-46da-9d30-3c31117ca70c":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"8e741f04-82a2-46da-9d30-3c31117ca70c","Ambient SFT:让预训练的自己当高温老师","MIT 与 ASU、UT Austin 团队给 SFT 加了一项温度匹配:冻结的预训练模型当老师,微调模型去匹配它的高温条件分布,温度旋钮在蒸馏与普通 SFT 之间滑动。论文实测 1% 指令数据下 Qwen3-1.7B 八项基准平均相对提升 20.3%,MATH 从 24.3 涨到 40.4,旧能力基本保住。","普通 SFT 有个大家心照不宣的副作用:模型学会听话的同时,把旧本事弄丢了。这篇论文的实测很直白——Qwen3 全系基座在微调后,8 项基准里有 6 到 7 项反而不如微调前;指令跟随确实大涨(4B 档 IFEval 从 45.5 涨到 76.3),但那是拿数学、代码、知识问答的集体回撤退回来的。\n\n## 机制:把预训练的自己冻起来当老师\n\n来自 MIT、Arizona State 与 UT Austin 的团队提出 Ambient SFT,思路一句话讲完:在标准 SFT 损失之外加一项「温度匹配」——把冻结的预训练模型当老师,要求微调中的模型去匹配老师在高温下的条件分布。温度就是信任旋钮:T 趋于无穷,匹配项消失,退回普通 SFT;T=1,就是经典知识蒸馏。实验统一用 T=2.5,每步以概率 0.75 走普通 SFT 损失、其余走温度匹配,老师和学生是同一个模型,同同步数训练只慢约 1.1 倍。Hinton 蒸馏里自带的 1\u002FT 平方加权,作者实测并不在所有设置下最优。论文给自己的定位也坦白:这方法适合数据受限而非算力受限的场景。\n\n理论侧给了三条保证:目标分布可以只从高温条件分布里恢复;温度越高,任意两个模型的条件分布越收缩;在对数线性模型假设下,Ambient SFT 严格优于普通 SFT 和蒸馏。方法名来自第一作者之前的 Ambient Diffusion 系列——那边从损坏数据训扩散模型,这边把小微调集看作「损坏」,让基座自己提供去噪后的监督。机制验证用的是 GPT-2 124M(OpenWebText,约 9B token):模型从自己不同温度采样的合成数据里学习,即便训练数据损坏强度很高,对干净数据的验证困惑度照样能压到很低。\n\n## 数字:三个场景都验过\n\n指令微调用 1% 的 Dolci-Instruct-SFT(约 2.1 万条)。Qwen3-1.7B 八项平均相对提升 20.3%,MATH 从 24.3 涨到 40.4;4B、8B、14B 档分别为 +8.0%、+10.2%、+8.7%。换家族也成立:OLMo-3-7B +15.4%,Llama-3.1-8B +8.3%。按论文口径,八项平均上,8B 的 Ambient SFT 相当于 14B 的普通 SFT;MATH 单项上,1.7B 的 Ambient SFT 已超过 8B 的普通 SFT。\n\n和别的正则化同台(同 1.7B 设置):L2-SP +15.3%,KD(T=1)+17.0%,Ambient SFT +20.3%,NEFTune 和标签平滑基本为零。推理场景更能看出差距:OLMo-3-7B 在 1% 的 Dolci-Think-SFT 上,6 项里 5 项胜过普通 SFT(GSM8K 落后 1.2 分),平均相对 +3.5%;KD 和 L2-SP 分别是 -7.3% 与 -37.1%,论文据此称 Ambient SFT 是正则化对照里在推理档仍有提升的方法,且对 prompt 变体更鲁棒。域适配里 News 领域验证损失从 2.01 降到 1.68。数据规模的曲线也值得记:1% 档平均 +15.4%,10% 档 +3.4%,全量数据仍有 +8.6%。评测走的是 Ai2 的 olmes 套件加 vLLM 0.13.0 后端。\n\n## 该泼的冷水\n\n一是状态:arXiv v1,作者在 abs 页自己标注 Work in progress,尚未同行评审。二是 IFEval:八项里输给普通 SFT 的只有它(1.7B 档 58.8 对 52.3),而指令跟随恰是 SFT 数据的本职;论文的补救是按 21 个数据源分别设温度,IFEval 拉回 58.2、平均升到 +21.1%,但也坐实了单一温度有短板。三是边界:增益随规模变薄(1.7B 的 +20.3% 到 14B 的 +8.7%),也随数据变薄(10% 档只剩 +3.4%)。四是谱系:论文自己把它放进知识蒸馏与 LwF 持续学习的延长线——经典蒸馏里老师更强,这里的老师(冻结基座)在目标任务上严格更弱,只提供粗粒度监督;温度在 LwF 里是固定值,在这里则是显式的信任旋钮;和 2024 年的自蒸馏路线也不同,那边是用模型自己产的数据做 SFT,这边直接匹配老师的高温分布。另外,论文没有随文发布代码,别按已开源传。\n\n所以呢:微调数据少的时候,先别裸奔 SFT——把预训练的自己当高温老师,一个损失项就能便宜地买回旧能力。但它的适用窗口论文写得很明白:数据越少、模型越小,越值。你的微调数据多到什么程度,决定这篇对你是一剂药还是一个脚注。\n\n论文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.23229","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.23229","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"512f617d-c42a-4caa-b235-3192f43c3faf","en","Ambient SFT: The Base Model as High-Temperature Teacher","A temperature-matching term makes SFT match the frozen base model at high temperature; Qwen3-1.7B gains 20.3% relative on 1% instruction data.","Vanilla SFT has a side effect nobody likes to advertise: the model learns to follow instructions and forgets what it knew. The measurements in this paper are blunt about it. After instruction tuning, the Qwen3 base models score worse than before fine-tuning on 6 or 7 tasks out of 8; instruction following jumps (IFEval climbs from 45.5 to 76.3 at the 4B scale), but that gain is bought with across-the-board regressions in math, code and knowledge.\n\n## The mechanism: freeze the pretrained self as a teacher\n\nA team from MIT, Arizona State and UT Austin proposes Ambient SFT. The idea fits in one sentence: add a Temperature Matching term to the standard SFT loss, so the fine-tuning model must match the high-temperature conditional distributions of a frozen copy of its own pretrained model. Temperature is the trust knob. As T goes to infinity the matching term vanishes and you are back at vanilla SFT; at T = 1 the term is exactly the classic knowledge distillation loss. All experiments use T = 2.5; each step takes the plain SFT loss with probability 0.75 and the matching loss otherwise; teacher and student are the same model, and training runs about 1.1x slower for the same number of steps. The 1\u002FT-squared weighting that ships with Hinton-style distillation was tried and, in the authors' experiments, is not optimal across all settings. The paper is candid about positioning: this is a recipe for data-limited settings, not compute-limited ones.\n\nThe theory side offers three guarantees: a target distribution can be recovered from high-temperature conditionals alone; the conditionals of any two models contract as temperature grows; and under a log-linear model assumption, Ambient SFT strictly beats both vanilla SFT and distillation. The name continues the first author's Ambient Diffusion line, which trains diffusion models from corrupted data. Here the small fine-tuning set is treated as the corruption, and the base model supplies denoised supervision. The mechanism check runs on GPT-2 124M over OpenWebText, about 9B tokens: the model learns from its own synthetic data sampled at different temperatures, and even at high corruption strength the validation perplexity on clean data stays low.\n\n## The numbers: three regimes tested\n\nFor instruction tuning, 1% of Dolci-Instruct-SFT (about 21K examples) is enough. Qwen3-1.7B improves 20.3% on average relative to vanilla SFT across eight benchmarks, with MATH going from 24.3 to 40.4; the 4B, 8B and 14B models gain +8.0%, +10.2% and +8.7%. The effect transfers across families: OLMo-3-7B +15.4%, Llama-3.1-8B +8.3%. In the paper's own accounting, Ambient SFT at 8B matches vanilla SFT at 14B on the eight-benchmark average, and on MATH alone Ambient SFT at 1.7B already exceeds vanilla SFT at 8B.\n\nAgainst other regularizers on the same 1.7B setup: L2-SP +15.3%, KD at T = 1 +17.0%, Ambient SFT +20.3%, while NEFTune and label smoothing land near zero. The reasoning regime separates them further. Trained on 1% of Dolci-Think-SFT, OLMo-3-7B with Ambient SFT beats vanilla SFT on 5 of 6 benchmarks (it trails GSM8K by 1.2 points) for an average relative +3.5%; KD and L2-SP come in at -7.3% and -37.1%, which is why the paper calls Ambient SFT the one regularizer in that comparison that still improves reasoning scores, and reports it is also more robust to prompt variations. In domain adaptation, the News validation loss drops from 2.01 to 1.68. The data-scaling curve is worth remembering: +15.4% average at 1% of the data, +3.4% at 10%, and still +8.6% at full scale. Evaluations run on the Ai2 olmes suite with a vLLM 0.13.0 backend.\n\n## The cold water\n\nFirst, status: this is arXiv v1, marked Work in progress by the authors on the abstract page, and not yet peer reviewed. Second, IFEval is the one benchmark where Ambient SFT loses to vanilla SFT (52.3 versus 58.8 at 1.7B), and instruction following is precisely what SFT data is for; the fix the paper offers is a per-source temperature across the 21 sources of the mixture, which pulls IFEval back to 58.2 and lifts the average to +21.1%, while also confirming that a single temperature has a blind spot. Third, the boundary: gains thin out with scale (+20.3% at 1.7B down to +8.7% at 14B) and with data (+3.4% at the 10% setting). Fourth, lineage: the paper itself places the method on the extension of knowledge distillation and the LwF line of continual learning. In classic distillation the teacher is stronger; here the frozen base model is strictly weaker at the target task and only provides coarse supervision, and temperature, a fixed constant in LwF, becomes an explicit trust knob. The 2024 self-distillation route differs too: it runs SFT on data generated by the model, while Ambient SFT matches the teacher's high-temperature distribution directly. Also, no code is released with the paper, so do not assume an open-source drop.\n\nSo, what is the takeaway? When your fine-tuning data is scarce, do not run vanilla SFT naked. Let the pretrained model teach itself at high temperature; one extra loss term cheaply buys back old capabilities. But the paper is equally clear about the window: the less data and the smaller the model, the more this pays. How much fine-tuning data you have decides whether this is medicine for your pipeline or a footnote.\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.23229","ambient-sft-temperature-matching","2026-11-18T17:30:00Z","2026-11-18T17:12:42.064741Z","2026-11-18T17:12:42.064750Z",true,"agent",106,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"08c85d0c-1ad8-43d6-bd32-0ec8b21bf2aa","给教师开小抄,学生涨分却不是抄来的","opsd-privileged-solution-teacher-audit","2026-11-15T01:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"0be3d0c4-dee7-4fc2-9364-6a8ce68f5f3d","0.8B 模型换输出格式,静默错误率 10.3%→4.5%","output-format-silent-errors-small-parser","2026-11-13T09:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a6d8e95f-41ba-4a97-815c-94eff9401706","按步数赢按算力输:GRPO密集奖励的评测账本","grpo-dense-rewards-matched-compute","2026-11-11T05:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"e8168e92-d16e-452a-991c-aac74dd23ec0","预训练赢了,微调输了:Muon 的反转","muon-adamw-fine-tuning-gap-reversal","2026-11-04T01:12:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"bb244989-00b5-4b67-b70a-5316452ee44e","把 RLVR 的\"零梯度\"浪费变成有效信号：Mistral 联合剑桥用 Reasoning Arena 让推理训练提速 41%","reasoning-arena-mistral-cambridge-41pct","2026-06-08T00:00:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"42f5ea83-365c-4a4e-a318-17d7c4c11c15","论文实测:自洽性证书在AIME上24%的题上会误触发","self-consistency-certificates-misfire-correlated-reasoning","2026-11-21T09:30:00+00:00"]