Classical Chinese poetry has long been a comfort zone for large language models. On existing benchmarks, flagship models like Qwen3-Max and Gemini-3-Pro score consistently high when judging Tang and Song verse. A new paper in the ACL 2026 long-paper proceedings raises an awkward question: do these models actually understand poetry, or have they merely memorized it?

The researchers built Neo-Classic, a benchmark designed to evaluate linguistic-aesthetic reasoning — the ability to reason genuinely about meter and discourse structure rather than retrieve from memory. The paper is available as arXiv:2609.19154 and appears in the Proceedings of the 64th Annual Meeting of the ACL (Volume 1: Long Papers, pages 27442-27465).

How the Test Is Built: Defeating Memorization

Most legacy benchmarks rely on verification or generation over historical corpora, so a model can answer by virtue of having seen the material during pretraining. Neo-Classic takes a different route: it combines a constructionist out-of-sample dataset with a suite of reverse-understanding probes. Every poem in the dataset is strictly metrical verse authored by contemporary experts, which is naturally disjoint from training corpora and blocks the direct-retrieval shortcut. Five behavioral probes layer constraints on top of one another to test hierarchical constraint satisfaction.

The Scorecard: A Collective Plunge

The team evaluated three flagship models: Qwen3-Max, Gemini-3-Pro, and DeepSeek-V3.2. Two levels of degradation emerged. First, when the same task switched from historical to contemporary texts, performance dropped by 20 to 50 percent. Second, on discourse-level ordering tasks, standard accuracy ranged from 0 to 13 percent. With expert-level guidance, reasoning-enhanced models climbed to 36 percent — still a notable gap from human experts.

Local Patterns Yes, Global Planning No

The authors' verdict is blunt: current LLMs capture local formal patterns but struggle with the global hierarchical planning required for robust linguistic-aesthetic reasoning. In other words, a model can match the meter of a single couplet, yet fail to coordinate the structural arc of an entire poem.

So What

The real contribution here is not the headline that LLMs are bad at classical poetry, but the methodology: constructionist out-of-sample data plus reverse probes separate "memory" from "reasoning" and measure each independently. For Chinese-language evaluation this matters especially — any genre densely represented in the training corpus can earn high scores as interest on memorization. Only when models can reason over texts no one has written before should we call it genuine understanding. For benchmark designers, this is an exam-writing playbook worth copying.

Reference: arXiv:2609.19154 / ACL 2026 Long Papers DOI