There's a long-overlooked gap between LLM training data and deployment scenarios: training data is mostly single-turn text completion tasks, while actual deployment is dominated by multi-turn conversation. The ICLR 2026 outstanding paper "LLMs Get Lost In Multi-Turn Conversation," through large-scale simulation experiments, systematically reveals the severity of this problem — this may be the strongest takedown of LLM practical capability to date.
Core finding: 39% average drop
The research team designed a scalable evaluation method for multi-turn dialogue capability, comparing model performance in single-turn vs multi-turn scenarios across six generation tasks. The results are sobering: current top open-source and closed-source models show significant performance drops in multi-turn dialogue, with average performance decline of 39%.
More noteworthy, the research further decomposes performance decay into two dimensions: a slight decline in capability, and a sharp decline in reliability — meaning the model not only answers worse, but also fails more often. In other words, users face not just quality decline, but a comprehensive rise in trust cost.
Problem root: premature commitment, unable to correct
After analyzing over 200,000 simulated dialogue turns, the researchers found LLMs make premature assumptions in early dialogue turns and rush to generate final answers, then over-rely on these early assumptions — once the direction is wrong, they can't correct. In the researchers' words: when LLMs take a wrong step in conversation, they get lost and can never come back.
This problem is directly tied to a core bias in the current model evaluation system: nearly all mainstream model capability evaluations are centered on single-turn, fully specified tasks, while users' real use scenarios often need multi-turn clarification and progressive requirement specification. There's a systematic bias between the two scenarios, and model benchmark scores can't truly reflect user experience in multi-turn interaction.
Warning to the industry
The significance of this research goes beyond problem identification — it provides a scalable diagnostic method, making multi-turn conversation capability evaluation possible rather than subjective impression. For model developers, this means reconsidering training data composition, introducing more multi-turn dialogue samples rather than all single-turn completion data; for evaluation frameworks, the dominance of single-turn benchmarks needs to be broken, with multi-turn scenarios deserving their own evaluation track; for application developers, more robust human-machine collaboration mechanisms need to be designed in key scenarios, giving users enough correction opportunities when models get lost, rather than letting them go all the way wrong.
Relying on parameter scaling and single-turn benchmark scores to evaluate models is no longer enough. Multi-turn conversation capability is becoming the key shortcoming for LLMs transitioning from toys to tools. The 39% number deserves every practitioner's attention.