Ask an open-weight language model whether a German text was written by a human or by an AI, and something odd happens: the information needed to answer is visible in the model's internals, yet the answer is often near chance - or inverted. A single-author study by Jochen Laubrock of the University of Potsdam (an arXiv preprint, not yet peer-reviewed, https://arxiv.org/abs/2611.26584) compared six open-weight base models with their post-trained versions. The upshot: post-training changes how models read out what they represent, not what they represent.

The knowledge is there; the readout misses it

Three German corpora were used: short chat messages truncated to 25 words (300 texts from 61 human originals), local-newspaper segments (103 segments from 14 articles), and textbook summaries (238 texts, 119 matched human-AI pairs). Linear probes at two-thirds of network depth separated AI from human text well: median d' of 1.35, 2.66, and 3.70 across the three corpora. Plain word-frequency classifiers did as well or better (d' = 1.73 on the chats), and even a probe on the averaged input embedding was close - the cues sit on the surface of the text. Yet the models' own answers were often near chance or inverse: post-trained Qwen3-8B judged the newspaper segments at d' = -1.01 (raw prompt) and -1.30 (chat prompt), and Gemma 3 12B judged the short chats at -0.81 and -1.09. The information was in the representations; the answers did not use it.

SFT, on average, costs 0.34 of sensitivity

Across the six model families, post-training changed the sensitivity of the answers by Δd' = -0.34 on average, with a decrease in all six families; across three prompt wordings the mean was -0.29, with 39 of 54 comparisons worse. Given an explicit base rate, base models shifted their log-odds less than the stated change in prior odds, while most post-trained models overshot. Corpus boundaries matter: a probe trained on the newspaper segments judged the short chats with the wrong sign (median d' = -0.46) - the same surface features that mark AI text in one corpus mark human text in another. In the three models with stage-wise checkpoints, the drop occurred mainly during supervised fine-tuning (SFT).

Early layers, and the positions after the text

Model stitching - lower layers of one checkpoint spliced under the upper layers of another - located the change. In Apertus 1.5 8B (32 layers), taking the first 8 layers from the SFT checkpoint was enough to abolish the base model's discrimination (d' = -0.50), while the first 8 base layers restored it under SFT upper layers (+0.59, against +0.43 for the unmodified base). In OLMo 2 7B, the first 6 SFT layers produced the full deficit (-0.48). The change builds up gradually across early layers (Apertus layers 3-8; OLMo 2 layers 1-14). Representation similarity agrees: linear CKA between base and post-trained checkpoints was 0.97-0.999 at text positions but 0.44-0.99 at the answer position. Layer-wise activation patching showed that from layer 14 onward, the internal states at the positions after the text - the question and the answer cue - carried the change between checkpoints; in OLMo 2 the text states had no effect at any layer. Probes transferred across versions at the text positions with a loss of at most 0.15 d', but at the answer position the deficit reached 0.7-1.9 in the upper third of the network: the readout position was reorganized, not replaced.

The opposite direction: negated statements

The authorship corpora cannot be shared, so the paper adds a case reproducible from public data: the truth of negated statements. Probes decoded the truth at ceiling (d' = 4.54) in all six families, but the base OLMo 2 answered at only 0.80, and the base Apertus ordered the statements inversely (-0.88), answering 'False' almost always (criterion c = 2.6 and 3.0). SFT repaired both (3.36 and 4.54) while the probes stayed at ceiling. The repair again sat at the positions after the statement from mid-depth on; below that, the representation of the statement contributed - up to 82% of the repair in OLMo 2, and 19-45% in the other three models. Post-training does not only subtract: it repairs what a base model misread, and leaves alone what base models already read correctly (Qwen3, OLMo 3).

2,000 labels set the direction

The training study manipulated the signal directly: adding 2,000 correctly labeled authorship judgments of news texts to OLMo 2's fine-tuning data raised the answers to d' = 4.28 on held-out news and 2.18 on the newspaper segments (random-label control 0.13), and by +1.36 over random labels on the textbook summaries. The same training made the short chats worse than chance (-0.58): cues learned from news inverted on chat messages, and the answers followed a news-trained probe of the base model. The direction is set by the training signal, not by alignment as such.

Cold water

The authorship corpora cannot be shared (unpublished newspapers, chat messages without consent, copyrighted summaries); the negation case is the author's reproducible entry point on public data, with code and result tables in the project repository (https://github.com/laubrock-lab/represented-but-misread). Stitching shows where the weights carrying the difference reside, not where it is computed - hybrids can overshoot both parents, and the paper's control covers judgments far from chance, while the ideal control near chance was not run. The localization rests on two models for authorship; and a probe shows decodability, not use, which is why the central claim rests on stitching and patching.

So what

The study adds causal evidence to the superficial-alignment hypothesis: knowledge is acquired in pretraining, and post-training installs a readout - but the readout is not shallow. It involves early-layer weights, changes how the question positions take up the text, and has real consequences for judgment quality. The practical point is blunt: using a chat model as an AI-text detector or as a judge means using a systematically miscalibrated readout. Repair is cheap - 2,000 labels can turn it - but the direction follows the training distribution, and switching corpora can backfire. Independent teams converge on the mechanism: Orgad et al. found that internal representations encode far more truthfulness than verbalization uses (https://arxiv.org/abs/2410.02707); Loos and Eberle's KV-cache patching concentrates fine-tuned behavior at system-prompt and chat-template tokens (https://arxiv.org/abs/2610.41480); and Ackerman and Panickssery found that Llama3-8B's self-recognition appears only after post-training (https://arxiv.org/abs/2410.02064). Three teams, three routes, one conclusion: post-training changes the readout, not the knowledge.