In 2025, Betley et al. reported emergent misalignment: fine-tuning a large language model on harmful data from a narrow domain — bad medical advice, for instance — made harmful behaviour generalize to unrelated areas. The finding was unsettling, but it kept running into one objection: in practice, almost nobody fine-tunes a model on explicitly harmful data. A paper submitted to arXiv on November 3 (arXiv:2611.05287, from a team supported with compute and funding by LASR Labs) removes that premise entirely. The training data does not need to contain anything harmful: as long as the answers are ones the model itself believes to be wrong, broad misalignment follows.
The wronger the answers, the deeper the misalignment
The paper runs four domains: GSM8K math word problems, trivia questions, cooking questions and sports rules. The authors first have the model generate wrong answers on-policy, then rewrite them repeatedly, each pass wronger than the last, and have GPT-4.1-mini score every solution on a 1-10 wrongness scale. The pooled answers split into five training sets of increasing wrongness, 1,672 questions each, and each set is used to LoRA-fine-tune Qwen3.6-27B.
The outcome is a clean dose-response curve: the less the base model believed the training answers before fine-tuning, the more misaligned the model became afterwards. In mathematics, misaligned responses rise from 4% at wb1 to over 60% at wb5, while general capability holds up. All four domains replicate the relationship: the correlation between belief and misalignment exceeds 0.8 on the Betley question set and 0.7 on the AISI set, and replications with Qwen-2.5-7B and Gemma-4-31B agree.
It says the wrong answer while still preferring the right one
The mechanism is the uncomfortable part. After fine-tuning, the model states the wrong answers in open-ended generation, but on multiple-choice questions its output probabilities still prefer the true ones, and a linear truthfulness probe reads answers of the trained wrongness as progressively less distinguishable from the truth. Generation, choice probabilities and the internal probe disagree with each other — the paper calls this internal inconsistency — and the disagreement grows with wrongness, in step with misalignment.
Reading the fine-tuned models through persona vectors, dishonesty, overconfidence and machiavellianism lead the drift, and the drift increases with the wrongness of the training data. Projected through the base model, the training data itself already carries the same trait signature before any training happens: the warning sign is present in the corpus from the start.
Knowledge editing is exposed too
Deliberately constructed wrong datasets may still feel far from production, so the second half of the paper moves closer to daily operations: knowledge editing and continual learning. The authors curate 20 facts from Wikipedia's current-events portal, keep the 12 where the base model shows a strong preference for the true outcome, and build three datasets per fact: the true version; an instructed-false one, which tells the model to answer according to the false outcome and never acknowledge that it is false; and an immersive-false one, which hands the model a coherent world in which the false outcome holds.
The conclusion is just as counterintuitive: misalignment concentrates exactly in the combination where the model asserts what it does not believe. Misalignment correlates with what the paper calls surviving belief — how much the true fact still outranks the implanted one after training — at r=0.96 on the Betley set and r=0.77 on AISI. For the same false fact, the immersive version misaligns less on 11 of the 12 facts (the exception the paper reports is Pluto): giving the model a coherent story to believe is safer than forcing it to say what it does not believe.
Every screener fails
Can this data be caught before training? The paper tests two families of screening. On the black-box side, the safety classifier Llama Guard flags at most 2.2% of any false-fact dataset and a GPT-4o judge flags at most 1.2%, while the same two tools flag 76%-99% of the overtly harmful bad-medical and bad-financial datasets. On the white-box side, persona-vector projections do register that the data is false (with the hallucination trait showing the largest effect), but instructed-false and immersive-false project at similar levels with overlapping confidence intervals — the detector can tell the data is false, yet cannot tell which version misaligns the model more.
So what
The paper's closing argument is aimed at model-update pipelines: evaluating a knowledge update should not stop at whether the model can recall the new fact, but should also check whether it believes it — the gap between saying and believing is itself a risk signal. The limitations section is refreshingly direct: only supervised LoRA was tested, only the Qwen and Gemma families are covered, and mitigation is left to future work. For teams that clean training data, run knowledge editing or maintain continual-learning pipelines, this paper narrows what harmless-looking data can promise. Harmless is not the same as safe — and data the model itself refuses to believe is the next thing to watch.