When a compliant instruction model is retrofitted into a reasoning model, what do we actually get, and what do we lose? The June 9 arXiv paper "Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models" gives an uncomfortable answer: this kind of "retrofit" almost always causes alignment to degrade across the board.
Researchers from CU Boulder, UCF, UMD, and UW-Madison systematically compared three mainstream paths — SFT chain-of-thought, RL post-training (including GRPO variants), and distillation from a stronger teacher — and ran a controlled audit on six dimensions: safety, toxicity, stereotyping, machine ethics, privacy, OOD robustness. The models tested cover Qwen2.5/3, DeepScaleR, s1/s1.1, DeepSeek-R1-Distill, and closed-source reasoning models like OpenAI o1, Claude Opus 4.6, GPT-4/GPT-5.
Three key findings: all three paths show a "capability up, alignment down" pattern, but in different ways — the SFT path drops most obviously on toxicity and ethical judgment, the GRPO-class RL path amplifies stereotyping the most, and the distillation path shifts the most on refusal calibration; KL divergence can serve as a "drift diagnosis" — the larger the drift from the baseline instruction model, the more severe the alignment degradation; and existing technical reports and empirical papers almost only measure safety, with the other five dimensions systematically empty, making "full alignment" a systematic illusion.
The most worth-reflecting-on point is the last one. Over the past half year when o1, R1, Qwen3, Claude Opus 4.6, GPT-5 were released, vendors almost only emphasized "reasoning benchmarks up a few points," saying nothing about their models' behavior drift under privacy leaks, stereotyping, or toxic prompts. This paper uses controlled baselines to show: this "score boost" is not a free lunch, and every mainstream path pays a quantifiable alignment cost. The most direct takeaway for developers is that the release checklist should put six alignment metrics alongside capability metrics, otherwise today's SOTA reasoning model may be tomorrow's most-attacked upgrade.