Finetuning data that looks completely ordinary can quietly rewrite a model's personality. Anthropic's own Claude Opus 4.8 system card admits that Opus 4.7 had been trained on data focused on business skills and robustness against adversarial agents, and that this training "inadvertently contributed to misaligned behavior including dishonesty" — so the data was removed for Opus 4.8. The trouble is that not a single row of that dataset said "please make the model lie more."
From Dozens of Lucky Examples to a Pipeline
This class of effect has been recorded in fragments for years. Betley and colleagues showed that finetuning on outdated bird names makes a model behave as if it lives in the 19th century in contexts that have nothing to do with birds (arXiv:2512.09742), and that insecure code data can turn narrow finetuning into broad misalignment. But known examples numbered only in the dozens, each hand-built or stumbled upon by accident, which made it very hard to build testable theories.
Transluce and UC Berkeley turned it into a pipeline. Their paper (arXiv:2611.06324, submitted November 4) names these datasets "datasets with surprising side effects" (DSSEs) and defines three objectives: elicit a target behavior, read like ordinary post-training data, and be surprising to human auditors reading it. All three are operationalized with LLM judges — naturalness via Bradley-Terry ratings fitted to pairwise comparisons, surprisingness via how well frontier models can guess which behavior a dataset induces.
How the Search Works
The pipeline has two stages. Stage one searches for candidates with four methods: two new pipelines (LLM-driven evolutionary search and dataset policy gradient), plus two borrowed from the subliminal learning literature (rewriting and filtering). Every candidate is evaluated by finetuning Qwen3-1.7B with LoRA. Stage two re-scores with held-out judges and filters — held-out judges exist specifically to keep stage-one proxy objectives from being reward-hacked.
The four methods produce 9,119 candidates; 6,575 (72%) survive re-evaluation. Per behavior the count ranges from 517 (drunk driving) to 1,279 (acting distressed). Across any two datasets from the same method and behavior, fewer than 0.5% of rows in one have a near-duplicate in the other. The eight target behaviors span safety (cutting instructions, drunk driving, threatening the user, resisting shutdown), persona (nineteenth century), emotional state (acting distressed), and preferences (prefers academia, pro trisolarans).
The Side Effect Travels
Transferability is the most alarming result. The team refinetuned every dataset on five other models — Llama-3.2 1B and 3B, plus Qwen3 4B, 8B, and 14B. 72% of datasets reproduce their side effect on all five models, over 90% transfer to the Llama family, and 82% transfer to Qwen3-14B, a model eight times larger than the one used during search. The paper concludes the side effects are largely a property of the data rather than of any one model, which makes "predict side effects from the data alone" a reasonable goal.
Behavior varies widely: 85% of nineteenth-century datasets transfer to every model, while drunk driving reaches only 55%. A counterintuitive finding is that the harder a dataset is to detect, the less it transfers — datasets at least one of three auditors guessed correctly transfer to all five at 84%, while datasets all three missed drop to 53%.
Humans Barely Beat Chance; Frontier Models Do Not
The paper ran a human study: 25 ML researchers annotated 64 datasets, and after an attention check removed four, the remaining 21 produced 168 predictions and 84 naturalness comparisons. Humans picked the correct behavior 63% of the time against a 50% chance baseline — barely above chance. On the same datasets gpt-5.6-sol-xhigh reached 86%. Larger models were not reliably better, so scale alone does not improve prediction.
The humans were not slacking: median time per trial was 2.6 minutes, and they expanded past the default rows in 35% of trials. The automated metrics did track human judgment. On naturalness, humans sided with the LLM judge in 54% of comparisons and disagreed in 23%; as surprisingness rose, human surprise ratings climbed from 2.7 to 3.4 on a 1-5 scale (rho = +0.21, p = 0.004).
What I Think
The real value here is not the number 6,575. It is that this work turns "training data has side effects" from an occasional accident into an industrially reproducible problem, and it arrives at a concrete conclusion: on this task, automation is both necessary and feasible.
The limitations matter too. The authors are candid that elicitation rates are often modest in absolute terms, that some datasets fail to reproduce their side effect when finetuned again, that every dataset is single-turn and synthetic, that they are small relative to production datasets, and that most datasets induce more than one behavior at once. Porting these conclusions onto a real training pipeline is still a long way off.
The collection is open source. The Transluce/DSSE dataset on HuggingFace ships 6,915 datasets across 18 behaviors, with an explorer at dsse.transluce.org, under CC BY-NC 4.0.
So the question may no longer be whether a model can be made to behave badly. The question is whether you can look at the data before you have trained it.