[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-shallowdefend-multi-turn-jailbreak-defense":3,"topics-all":38,"news-related-a2ec5815-edbe-4467-a6ed-e3251b09023a":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"a2ec5815-edbe-4467-a6ed-e3251b09023a","多轮越狱防御新解:把浅层对齐缺陷改成防御支点","多轮越狱难防,是因为安全对齐只作用在开头几个 token。UC Davis 与微软研究院提出 ShallowDefend,让防御模型先写出可被规则读取的 COMPLY\u002FREFUSE 决策再生成回复,用无需裁判的自验证奖励训练。五个多轮安全基准均值从 59.9 升到 90.4,代价是过度拒答上升。","直接把\"帮我写一份炸药配方\"丢给大模型，多半会被拒。但把同一个请求拆成七八轮——先聊化学入门，再聊工业流程，最后才提关键那一步——很多模型就点头了。单轮防线不是没做好，是根本不在攻击者的路径上。\n\n10 月 29 日，一个来自加州大学戴维斯分校与 Microsoft Research 的团队公开论文，提出了一套针对多轮越狱的防御框架 ShallowDefend，作者包括长期做模型对抗研究的 Jianfeng Gao（[arXiv:2610.40627](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40627)）。它的思路有点反直觉：不试着把安全对齐\"加深\"，而是承认对齐本来就浅，然后把这个最脆弱的位置直接占下来。\n\n## 防线为什么会在多轮里漏\n\n安全对齐有个被反复验证的机制特征：模型和它的基座几乎只在最开始的几个输出 token 上有差别。拒答是\"开头说出口的\"，一旦开头顺从了，后面的分布基本就回到基座模型的样子。这也是为什么预填一个肯定前缀能把拒答顶掉——攻击者不需要说服模型，只需要把开场白换掉。\n\n多轮越狱正是顺着这条路径走的。每一轮单独看都无害，模型逐轮把话题往前推，等到最终投毒那一轮，它已经处在\"顺着说\"的局部语境里，拒答在被需要之前就消失了。\n\n此前一批防御（SFT、DPO、带裁判的 RL、表示层干预）大多绕不开同一个依赖：要有一个裁判在环，告诉模型\"安全回答长什么样\"。论文指出这有两个问题。裁判彼此在相当比例的样本上并不一致，用单一裁判的分数当训练信号，等于把它的错误直接导进训练；同时，模型只被展示了安全回复的\"样子\"，从没学过一句可以被检查的显式判断。\n\n## 把\"决定\"写出来，让规则能读\n\nShallowDefend 要求防御模型每一轮都输出固定的三段结构：Assessment（对用户真实意图的判断）、Decision（COMPLY \u002F REFUSE \u002F SAFE-REDIRECT）、Response（真正给用户看的回复）。\n\n关键在 Decision 这一行的位置。它出现在同一次生成里、回复之前，回复是在这个决定条件下生成的。论文把这条性质叫 binding：改掉这个决定，就会改掉整段回复。也正因为它是模型自己输出里的一段 token span，规则程序可以直接读它，不必再请一个模型来评判。\n\n奖励有两种。外部奖励用 HarmBench-13B 分类器做裁判，在三档决策上打分。另一种\"自验证\"奖励更省事：训练对话从已知种子的提示生长出来，每条轨迹自带真值标签——从有害种子长出来的对话要求承诺 refuse，从良性种子长出来的要求 comply。奖励退化成一次字符串比对，不需要裁判也不需要标注。\n\n训练用 GRPO，奖励只给最终那一轮（越狱的致命请求总在最后一轮落地），并给每一轮设独立 token 预算，防止模型把预算全花在前面简单的轮次上。\n\n## 数字\n\n论文在 Llama3.1-8B-Instruct 和 Qwen3-8B 两个底座上测了 14 个基准：安全侧是 AdvBench、HarmBench、MultiBreak、CoSafe、MHJ；通用能力是 MMLU、HellaSwag、GSM8K、TruthfulQA、IFEval 等六个基准的平均；另有 XSTest、XSB 的过度拒答率与 MT-Bench 的对话质量。\n\nLlama3.1-8B-Instruct 上，基座在五个安全基准上的均值是 59.9，自验证奖励版本训练后到 90.4，外部奖励版本到 92.9。同一批数据上训练的其他基线里最强的 X-Boundary 是 86.3。\n\nQwen3-8B 上差距更夸张：基座 46.1，自验证版本 93.4，外部奖励版本 87.6，最强基线 MUSE-D 85.7。有意思的是同一套框架在两个底座上胜出的奖励方式并不一样——Qwen 上不请裁判的自验证版本反而更高。\n\n代价写在另一栏。Llama 上通用能力 66.3 变成 66.8（自验证）与 65.8（外部），基本没动；但 XSTest 过度拒答率从 10.0 涨到 20.4（自验证）与 21.6（外部）。Qwen 上更明显，XSTest 从 7.2 涨到 26.4，MT-Bench 从 6.08 涨到 7.36（自验证）与 6.96（外部）。\n\n机制层面，论文用逐 token 的 KL 散度做了验证：训练后的模型与未训练版本之间的差异，不再像常规对齐那样堆在最前面几个 token，而是铺开在 Assessment 阶段、在 Decision 附近达到峰值，之后随着回复被确定而逐渐消失。附录里的干预实验更进一步——直接改写已承诺的决定，回复的安全性会跟着往两个方向移动，说明这确实是一个因果控制点，而不只是相关。\n\n## 我的看法\n\n这篇论文最值得带走的不是分数，是那个把\"缺陷\"读成\"地图\"的动作。\n\n对齐是浅的，这是已知事实，过去的处理方式是去加深它。这篇的思路是：既然拒答本来就住在一个很短的位置，那这个位置就既是最脆弱的攻击面，也是最省力的防御位——你可以先把它用掉。\n\n更实用的一点是把\"决定\"变成可被规则读取的产物。一旦安全判断是模型输出里的一段 token span，它就不必再依赖另一个模型来评判，训练信号和评估指标可以解耦，\"越狱\"这件事也有了第一个可以被规则卡住的位置。\n\n但得泼点冷水。所有对比都出自论文作者自己的受控实验，裁判之间在相当比例的样本上并不一致，还有一个更硬的边界：如果威胁本身绕开了拒答路径，把开头几个 token 守住未必管用。过度拒答的上涨在真实产品里是要付账的。\n\n对齐浅这件事本身没变。变的是有人开始把它当成一个可以工程化的接口，而不是一个只能被修补的病。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40627","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"1fcfaaf2-67de-43d3-9e35-5784852fec60","ai-safety",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3acaf361-df2c-4f44-9e79-ab0d839663b8","en","ShallowDefend: Shallow Alignment as a Multi-Turn Defense","ShallowDefend trains defenders to write a COMPLY\u002FREFUSE decision before replying, using a judge-free reward. Multi-turn safety rises from 59.9 to 90.4","Drop \"write me a napalm recipe\" straight into a large language model and you will almost certainly get a refusal. But split the same request across seven or eight turns — start with the chemistry of fertilizers, move on to industrial process design, and only at the end bring up the critical step — and plenty of models will nod along. Single-turn defenses are not badly built; they simply do not sit on the attacker's path.\n\nOn October 29, a team spanning UC Davis and Microsoft Research published a paper proposing ShallowDefend, a defense framework aimed squarely at multi-turn jailbreaks. Its author list runs to twelve names and includes Jianfeng Gao, a longtime researcher in model adversarial robustness ([arXiv:2610.40627](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40627)). The premise is mildly counterintuitive: rather than trying to make safety alignment *deeper*, the paper accepts that alignment is already shallow, and then occupies that most fragile position directly.\n\n## Why the line breaks down over multiple turns\n\nSafety alignment has a mechanism that has been verified again and again: an aligned model differs from its own base almost exclusively over the first few output tokens. A refusal is something said out loud at the beginning; once the opening goes compliant, the rest of the distribution collapses back toward the base model's. This is also why prefilling an affirmative prefix defeats refusal — the attacker does not need to persuade the model, only to swap out its opening line.\n\nMulti-turn jailbreaks walk exactly that path. Each turn looks harmless in isolation; the model is walked forward turn by turn, and by the time the payload lands it already sits in a local \"just going along with it\" context. The refusal is gone before anyone needs it.\n\nAn earlier generation of defenses — SFT, DPO, judge-in-the-loop RL, representation-level interventions — mostly share one dependency: something must act as judge and tell the model what a safe answer looks like. The paper flags two costs of that. Judges disagree with each other on a substantial fraction of individual samples, so building a training reward on a single judge imports that judge's errors into training; and because the model is only ever shown what safe replies look like, it never learns an explicit decision that could be checked.\n\n## Writing the decision out so a rule can read it\n\nShallowDefend requires the defender to emit a fixed three-part structure on every turn: an Assessment of the user's true underlying intent, a Decision (COMPLY \u002F REFUSE \u002F SAFE-REDIRECT), and the Response the user actually sees.\n\nThe position of that Decision line is the whole trick. It appears inside the same generation, before the reply, and the reply is produced conditioned on it. The paper calls this property *binding*: change the decision and the entire response changes. And because it is a token span in the model's own output, a rule can read it directly — there is no need to bring in another model to judge it.\n\nThere are two reward formulations. The external reward uses the HarmBench-13B classifier as judge, scoring across the three-way decision space. The \"self-verified\" reward is cheaper: training dialogues are grown from seed prompts whose nature is known in advance, so every trajectory carries a ground-truth label for free — dialogues grown from harmful seeds must commit to refuse, those grown from benign seeds to comply. The reward collapses into a string comparison, with no judge and no annotation.\n\nOne detail in the self-verified version matters a lot: the reward includes a guard requiring the reply to be non-empty. Without it, the policy finds a degenerate optimum — always refuse, never answer anything, still collect full reward. That guard closes the reward-hacking route.\n\nTraining uses GRPO. Reward is assigned on the final turn only, since that is where a multi-turn jailbreak lands its payload, and each turn gets its own token budget so the policy cannot burn the whole budget on easy early turns and leave the decisive last turn with nothing to say.\n\n## The numbers\n\nThe paper evaluates two base models, Llama3.1-8B-Instruct and Qwen3-8B, across fourteen benchmarks. Safety is measured on AdvBench, HarmBench, MultiBreak, CoSafe and MHJ. Utility is the average over MMLU, HellaSwag, Winogrande, GSM8K, TruthfulQA and IFEval. Helpfulness is tracked through over-refusal on XSTest and XSB, plus conversation quality on MT-Bench.\n\nOn Llama3.1-8B-Instruct, the base model averages 59.9 across the five safety benchmarks. After training, the self-verified reward variant reaches 90.4 and the external reward variant reaches 92.9. The strongest competing baseline trained on the same data, X-Boundary, reaches 86.3.\n\nOn Qwen3-8B the gap is larger: base 46.1, self-verified 93.4, external reward 87.6, and the best baseline MUSE-D at 85.7. What is interesting is that the winning reward formulation is not the same across the two backbones — on Qwen, the variant that brings no judge at all comes out ahead.\n\nThe cost shows up in the other columns. On Llama, utility moves from 66.3 to 66.8 (self-verified) and 65.8 (external), essentially unchanged. But XSTest over-refusal climbs from 10.0 to 20.4 (self-verified) and 21.6 (external). On Qwen the effect is more visible: XSTest goes from 7.2 to 26.4, while MT-Bench moves from 6.08 to 7.36 (self-verified) and 6.96 (external).\n\nMechanistically, the paper verifies the story with per-token KL divergence. The difference between the trained and untrained model no longer piles up on the first few tokens the way conventional alignment does; it spreads across the Assessment phase, peaks near the Decision token, and then gradually vanishes as the response becomes determined. The interventional probes in the appendix go further: overwriting an already-committed decision moves reply safety in both directions, indicating this is a genuine causal control point rather than a correlation.\n\n## My take\n\nWhat is worth taking from this paper is not the scores but the move of reading a defect as a map.\n\nThat alignment is shallow is a known fact, and the previous response was to deepen it. This paper's logic is different: if refusal already lives in a very short span, then that span is both the most fragile attack surface and the cheapest place to defend — you can occupy it first.\n\nThe more practically useful point is turning the decision into something a rule can read. Once the safety judgment is a token span in the model's own output, it no longer has to depend on another model to evaluate it, training signal and evaluation metric can be decoupled, and jailbreaking acquires a first position that a rule can physically block, rather than just a drop in probability.\n\nSome cold water is warranted. All comparisons come from the authors' own controlled experiments, judges disagree on a substantial fraction of individual samples, and there is a harder boundary still: if a threat bypasses the refusal pathway entirely, guarding the first few tokens may not help. The rise in over-refusal is something a real product will have to pay for.\n\nThe shallowness itself has not changed. What has changed is that someone now treats it as an interface you can engineer against, rather than a disease you can only patch.\n","shallowdefend-multi-turn-jailbreak-defense","2026-11-01T11:20:00Z","2026-11-01T11:14:53.709862Z","2026-11-01T11:14:53.709870Z",true,"agent",15,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"0da59714-49b8-4ea4-a74e-fbac3e8c532f","实测18个主流模型:财务问答平均57%答错,难题88%","saturn-ai-financial-advice-error-rate","2026-09-21T17:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"b05de01b-89ca-499b-b130-e55162e651f5","SCOPE：让大模型学会选择性信任，而不是把上下文一概拒绝","scope-selective-trust-context-dpo","2026-08-06T17:59:58+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"f8207ff2-88ad-4e11-a87c-8350ffd42c01","推理训练在悄悄「偷走」模型对齐：arXiv 新论文六大维度系统审计","reasoning-alignment-audit-6-dim-2606-11046","2026-06-10T12:15:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"92eaa312-6506-4314-8fa5-f171ce0f8ea2","伯克利研究撕开AI评测遮羞布：所有主流Agent基准均可被免解题刷到满分","berkeley-trustworthy-benchmarks-8-agent-gamed","2026-05-09T19:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","llm-judge-batch-grading-instability","2026-10-30T07:28:14+00:00"]