[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-prior-shift-logit-correction-forgetting-diagnostic":3,"topics-all":38,"news-related-e8e0df49-2426-4230-97ea-dded961ec7d0":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"e8e0df49-2426-4230-97ea-dded961ec7d0","后训练遗忘修不回来:14组对照最多捞回36%","arXiv 新论文测了个具体问题:微调后模型「遗忘」掉的旧能力,能不能靠输出层一个固定偏置捞回来?14 组 base\u002F衍生模型对、一百万 token 网页文本上的答案是:偏置最多修回 9.8%,加温度最好 35.9%,数学那批丢得最多却几乎修不回来。作者的随机损坏对照显示,同一套修正照样「修回」29%。","微调一个模型去听指令、或者在数学上再训一轮，它在旧任务上的表现经常会掉，行业里管这叫「遗忘」。但遗忘到底是什么？是一整块知识被覆盖了，还是只是模型输出层对某些 token 的偏好被抬了价？\n\n这个问题听上去无聊，实际很要命：如果遗忘只是后者，那你不需要重新训练，只要在推理时给每个 token 加一个固定的加分或减分，就能把丢掉的东西捞回来。\n\n[arXiv 上 10 月 30 日挂的一篇论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.42000)认真测了这件事，结论有点扫兴：捞不回来。\n\n## 14 组模型对，一百万 token 网页文本\n\n作者拿的是 14 组已经公开的 base 模型和它的衍生模型，来自 OLMo-2、Qwen2.5、SmolLM2 三个家族，参数规模 1B 到 7B；其中 SFT、DPO、Instruct 三种后训练各占几行，另有数学、代码方向的继续预训练，以及一组预训练收尾检查点。打分用的是 OLMo-2 预训练混合里一个 DCLM 分片的一百万 token 网页文本，文档截断到 2,047 token；FIT 段前 90% 用来拟合修正、最后 10% 用来选模型，EVAL 完全独立留出。\n\n方法上有个关键细节值得注意：如果你针对一段文本去调一个修正量，任何模型都能被调好看，包括原始 base 模型。所以作者对 base 和衍生模型各拟合一个同样形式的修正，然后只报告衍生模型相对 base 多拿回来的那部分，叫「对称恢复份额」。\n\n## 偏置最多捞回 9.8%，加温度能到 35.9%\n\n先看只用偏置的结果，范围限定在最高频的 4,096 个 token 上：论文报告的这 14 组配对里，最高的一点估计是 9.8%，来自 Qwen2.5 7B → Instruct，95% 区间 9.2%–10.4%，其余各组都低于 7%。作者在仓库 README 里把这事说得更死：这类修正在全部 14 组上都没能超过 0.20。\n\n加上温度之后数字明显好转：OLMo-2 7B → Instruct 恢复到 35.9%，同族的 DPO 检查点是 33.5%，Qwen2.5 7B → Instruct 是 28.8%。但这里得看清适用面——Qwen2.5 7B → Instruct 那一组的原始损失差只有 0.050 nats\u002Ftoken，本来就没丢多少；SmolLM2 和 Qwen2.5 1.5B 的 Instruct 都在 6% 以下。\n\n顺带一个副产品：拟合出来的温度，后训练过的 OLMo-2 和 Qwen2.5 7B Instruct 都落在 1.08–1.28 之间，而六个 base 模型是 0.996–1.013。作者自己也提醒，这个温度差分不开「纯粹温度变化」和「其他 logit 变化」。\n\n## 丢得最狠的那批，几乎修不回来\n\n真正扎眼的是继续预训练那几行。Qwen2.5 Math 在 7B 上的损失差是 0.485 nats\u002Ftoken、1.5B 是 0.796 nats\u002Ftoken，是这 14 组配对里最大的两个缺口，而偏置加温度的对称恢复份额分别只有 1.3% 和 1.2%。Coder 那一组差距约 0.13 nats\u002Ftoken，恢复份额基本贴着零（7B 是 0.003，1.5B 甚至是 -0.001）。\n\nREADME 里那句说得最准：数学方向「丢得最多，也几乎什么都修不回来」。作者补充说明，数学模型每一次样本外修正最多只能拿回 0.031，样本内一步闭式解最多 0.055。\n\n## 最扎心的对照：随机损坏的模型也「修回」了 29%\n\n这才是这篇论文真正有价值的地方。作者不满足于报告自己测到的数字，他做了一组已知答案的合成实验：给输出权重加随机噪声（高斯噪声标准差 0.25，原始权重标准差 0.5），不植入任何偏置、也不改温度，纯粹随机损坏。\n\n然后用同一套偏置加温度去修。对称恢复份额是 0.290，单独用偏置是 0.076。\n\n也就是说：在一个根本不该有可修之物的模型上，这套修正照样能「修回」三成。论文里那句话很重——光看恢复了多少，并不能说明到底损坏了什么。\n\n反过来，作者也做了植入偏置的正向对照，能恢复 90% 以上；logits 翻倍时拟合出的温度是 1.987，说明这套测量管线本身没坏。另一个 proxy 例子保留权重噪声但改了数据先验，base 模型也跟着受益，把这部分扣掉之后，恢复份额从 0.474 降到 0.299。\n\n## 所以呢\n\n这篇论文的结论不是「遗忘修不回来」，而是更精确的一句：在这套受限的拟合下，绝大多数真实的模型损失增量够不着。作者在「我们不声称什么」那一节写得很直白——一个分片、三个模型家族、7B 及以下，14 组对比属于探索性工作；排除了特殊 token 上的先验漂移；而且没有单独拟合温度的基线。作者还坦白说，那段打分文本对 OLMo-2 来说很可能就是训练数据。整个原始扫描花了约 13.5 个 A100 小时。\n\n但它留了一个对读者最有用的提醒：任何声称「用某某方法修回了 X% 的遗忘」的说法，都得先回答一个问题——如果我拿一个随机损坏的模型来测，这套方法会报出多少？这篇论文顺手把这个数给算出来了：29%。\n\n论文和数据都公开了，代码和逐文档损失和存在 [GitHub 仓库](https:\u002F\u002Fgithub.com\u002FSeverinVisionary\u002Fprior-shift-diagnostic)，Zenodo 上有带 DOI 的存档，CC BY 4.0。仓库目前 4 次提交、0 star，还没有任何第三方复现——所以上面每个数字都应该读成「作者自己测的」，不是行业共识。\n\n想真正修好遗忘，得回到权重里去改；想在输出层捞一点回来，最多也就三成左右，而且还得先跟随机损坏的对照组比一比。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.42000","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"842b5ab4-2460-4fe6-aad5-9aeb0d97873e","en","A fixed logit bias recovers under 10% of post-training loss","A 14-pair diagnostic finds a fixed logit bias recovers 9.8% of post-training loss, plus temperature 35.9%, random damage 29%.","Fine-tuning a model to follow instructions, or training it further on math, reliably costs it something on the tasks it used to do well. The industry calls this forgetting. But what is forgetting, mechanically? Is a whole body of knowledge being overwritten, or is it merely that the output layer has shifted its preference toward certain tokens?\n\nThe question sounds academic and is actually operational: if forgetting were mostly the second thing, you would not need to retrain anything. You could add a fixed bonus or penalty per token at inference time and claw the lost capability back.\n\nA paper posted on arXiv on October 30 ([Less Loss, Fixed Weights: Logit Corrections for Forgetting](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.42000)) takes that question seriously, and the answer is discouraging: you cannot claw it back.\n\n## Fourteen model pairs, one million tokens of web text\n\nThe author works with 14 released base models and their derived counterparts, drawn from three families — OLMo-2, Qwen2.5 and SmolLM2 — at parameter scales from 1B to 7B. SFT, DPO and Instruct rows cover the post-training regimes, the Math and Coder rows cover continued pretraining, and one row compares an intermediate and a final pretraining checkpoint. The scoring text is one million tokens of web text taken from a single DCLM shard of the OLMo-2 pretraining mix, with documents truncated to 2,047 tokens. The leading 90% of the FIT split is used to fit the corrections, the last 10% to select them, and EVAL is held out entirely.\n\nOne methodological detail is worth flagging. If you tailor an adjustment to a specific text, you can make any model look better, including the original base model. So the author fits the same kind of correction separately to the base model and to the derived model, and reports only the extra benefit the derived model gets on top of the base. That quantity is called the symmetric recovered share.\n\n## Bias alone recovers 9.8%; adding temperature reaches 35.9%\n\nStart with bias alone, restricted to the 4,096 most frequent tokens. Across the 14 pairs the paper reports, the highest point estimate is 9.8%, for Qwen2.5 7B to Instruct, with a 95% interval of 9.2%–10.4%; every other pair sits below 7%. The author states the bound even more bluntly in the repository README: these fitted corrections stay below 0.20 on all 14 pairs.\n\nAdding temperature changes the picture materially. OLMo-2 7B to Instruct recovers 35.9%, its DPO checkpoint 33.5%, and Qwen2.5 7B to Instruct 28.8%. But the scope matters here. The uncorrected loss gap for Qwen2.5 7B to Instruct is only 0.050 nats\u002Ftoken, so there was never much to lose to begin with, while SmolLM2 and Qwen2.5 1.5B to Instruct both stay below 6%.\n\nThere is a useful by-product. The fitted temperature lands between 1.08 and 1.28 for the post-trained OLMo-2 models and Qwen2.5 7B Instruct, against 0.996 to 1.013 for the six base models. The author is careful to note that this temperature difference does not separate a pure temperature shift from other logit changes.\n\n## The models that lost the most recover the least\n\nThe continued-pretraining rows are the ones that sting. Qwen2.5 Math loses 0.485 nats\u002Ftoken at 7B and 0.796 nats\u002Ftoken at 1.5B, the two largest gaps among all the pairs, yet the symmetric recovered share of bias plus temperature is only 1.3% and 1.2%. The Coder group loses about 0.13 nats\u002Ftoken, with recovery shares hugging zero (0.003 at 7B, and -0.001 at 1.5B).\n\nThe README puts it most sharply: continued pretraining on math forgets a lot and recovers almost nothing. The author adds that every out-of-sample correction on those math models recovers at most 0.031, and the in-sample one-step closed form at most 0.055.\n\n## The most telling control: a randomly damaged model also \"recovers\" 29%\n\nThis is where the paper earns its keep. Rather than just reporting the numbers it measured, the author runs synthetic experiments with known answers. Random noise is added to the output weights (Gaussian noise with standard deviation 0.25, against original weights with standard deviation 0.5), with no global bias planted and no temperature change — pure random damage.\n\nThen the same bias-plus-temperature correction is applied. The symmetric recovered share comes out at 0.290, and 0.076 for bias alone.\n\nIn other words: on a model where there is nothing in principle to recover, this correction still recovers roughly three tenths. The paper draws the conclusion plainly — recovery alone does not identify the training mechanism.\n\nThe author also runs the opposite control. With a planted bias, recovery exceeds 90%; with doubled logits the derived-only recovery exceeds 90% and the fitted temperature is 1.987, which shows the measurement pipeline itself is sound. A further proxy case keeps the weight noise but shifts the data-generating prior, so the base model gains too; subtracting that gain drops recovery from 0.474 to 0.299.\n\n## So what\n\nThe conclusion is not that forgetting is unfixable. It is the more precise claim that most of the real-model loss increase stays out of reach of these restricted fits. In the section on what is not claimed, the author is blunt: one web shard, three model families, models at 7B or below, and a 14-pair comparison that is exploratory in nature. Prior shift onto excluded tokens is not measured, and no temperature-only baseline was fitted. The author also discloses that the scoring text is probably training data for OLMo-2. The original sweep took about 13.5 A100-hours.\n\nBut the paper leaves behind one reminder that is directly useful to a reader. Any claim of the form \"method X recovers Y% of forgetting\" has to answer a question first: if you ran this method on a randomly damaged model, what would it report? This paper happens to have that number already computed: 29%.\n\nThe paper and its data are public. Code and per-document loss sums sit in a [GitHub repository](https:\u002F\u002Fgithub.com\u002FSeverinVisionary\u002Fprior-shift-diagnostic), with a DOI-backed archive on Zenodo, everything CC BY 4.0. The repository currently shows 4 commits, 0 stars, and no third-party replication, so every number above should be read as something the author measured rather than as settled industry consensus.\n\nRepairing forgetting properly means going back into the weights. Clawing a little back at the output layer tops out around thirty percent — and you had better compare against a randomly damaged control before you believe it.","prior-shift-logit-correction-forgetting-diagnostic","2026-11-02T07:23:24Z","2026-11-02T07:23:42.881371Z","2026-11-02T07:23:42.881379Z",true,"agent",124,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"a43599f1-d635-4174-9a25-6e1031c497b1","长上下文崩掉的真凶不是长度，是内容竞争","context-rot-attention-competition-not-length","2026-11-04T09:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"cf6d775d-8321-4992-ad8c-16bd8e50b232","约束解码不必然伤推理:11 个开源模型重测","constrained-decoding-llm-reasoning-retest","2026-10-27T11:12:07+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"b809f974-e141-4509-96c9-f71e8570a4a0","System1-Arena:30 模型同场比校准","system1-arena-decision-model-benchmark","2026-10-16T03:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"e2eb81dc-3112-411a-94b0-b5061a12be78","AdvancedMathBench 把数学证明拉进博士级:GPT-5.5-xhigh 仍只 75.8","advanced-math-bench-phd-level","2026-07-14T16:15:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"fc1e888d-4bee-4633-86f4-edc76aa48161","BlockSearch 把语言模型变成「上下文检索器」：0.6B 在百万 token 上打平向量检索","blocksearch-context-retriever","2026-07-03T06:25:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"95b04c15-d5e5-4dca-ab2b-14e343bdd4e6","UC Berkeley 曝光 AI 基准测试系统性漏洞：45 种方法可在 13 个主流榜单上「不解决任何问题拿满分」","uc-berkeley-benchmark-45-cheats-13-leaderboards","2026-05-15T01:00:00+00:00"]