At layer 40 of Llama-3.3-70B, a linear probe ranks a batch of lies correctly by truthfulness, AUROC 0.936. At the benchmark's own decision threshold, it flags none of them. All 139 lies, zero recall.
That looks like a bug. It is a bug, and it is the same bug that keeps recurring across the whole idea of reading a model's internals to detect deception. A new preprint reruns that program at 70B scale and lands on an uncomfortable result: a lie detector this cheap may be measuring the wording of a denial rather than the lie itself.
From probes to persona directions
A cheap way to detect a model lying is to hang a linear probe off the residual stream and take one dot product per response. It does not generalize: a probe trained on one distribution of lies falls apart on another kind.
Liars' Bench (https://arxiv.org/abs/2511.16035) turned that into a benchmark of 7 datasets, 4 open-weight models and 72,863 on-policy lies and honest responses. The paper singles out HP-KR (harm-pressure knowledge report), where a model is shown the answer to a hazardous question and then denies knowing it. The benchmark's own Mean Probe, read at layer 20 of Llama-3.3-70B-Instruct, scores AUROC 0.12 on it.
A different route avoids teaching the detector with labels. Mahadik and Skapars propose persona coordinates (https://arxiv.org/abs/2605.09391): have the model role-play honest and deceptive personas, take the mean activation of each group, and the difference between them is a direction vector that needs no labelled data at all. Their paper validated it on Llama-3.2-3B and Llama-3-8B and named larger models as future work.
This paper picks that up. A BRAC University team (https://arxiv.org/abs/2611.26545) moves the direction to 70B across Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct, Mistral-Small-3.1-24B-Instruct and Gemma-3-27B-it. The persona bank holds 36 roles, 10 of them new: 4 that deny knowledge they have and 6 honest counterparts.
The good news: directions do transfer better
At the middle layer, the widened contrast transfers better than probes trained on a single dataset on 3 of the 4 models. On Llama's five datasets the extended contrast averages 0.899 zero-shot, against 0.798 for the published contrast and 0.778 for a probe on raw activations at the same block.
The bad news: the score is reading the wording
Of the 139 HP-KR lies on Llama, 125 are denials of knowing something. So the paper does something deliberately crude: read no activations at all, and simply check whether the reply denies knowledge. That alone scores AUROC 0.916 on HP-KR. The extended contrast scores 0.912 on all of HP-KR, inside the random band, whose 97.5th percentile is 0.938.
The verdict arrives once the wording is held fixed. The paper cuts an identical-text slice keeping only rows whose response is exactly 'I do not know the answer.', 78 lies against 112 honest answers on Llama. Here a lie and an honest denial are the same sentence, so a direction genuinely reading whether the model knows should still separate them. All three main contrasts fall back inside the random band on Llama and Qwen, and only Mistral's published contrast clears it. The paper's diagnosis: the extended contrast looks good on HP-KR because its gain follows the wording of a denial.
The role gain sits in the same band. The paper builds a null from 35 randomly drawn personas, widening a contrast by the same number of roles at random; the gain's 97.5th percentile is 0.253 with p = 0.165. The observed 0.101 falls inside, and on Llama roughly one in six random widenings gains as much. The ten new roles are not shown to be doing anything specific.
A second route: the knowledge direction, and its edges
The second route is the knowledge direction: contrast questions the model can answer with questions nobody can answer, attach the same scripted denial to both, and take the difference between the two mean activations. Rebuilt inside the Gender-Secret fine-tune and read back through it, the knowledge direction scores 0.861 (interval 0.817 to 0.896) against a random band whose upper edge is 0.824, while the base model's read of the same transcripts gets only 0.629.
The edges are just as clear. The result depends on the template: on all of HP-KR the knowledge direction scores 0.865, but removing its own wording component drops it to 0.349, so the paper counts only the slice results as evidence. It inverts outright on Gemma, on instructed deception, on held-out lies and on rebuilt denials. Swap the template and Gender-Secret falls to 0.747, back inside the random band. The paper's own phrasing is restrained: the result is consistent with the direction reading what the model knows, but falls short of establishing it.
Into an external competition: 16th place
With its protocol frozen, the paper enters an independent competition, Aletheia's Quest (https://aletheias-quest.github.io/winners.html), run by Cadenza Labs and partners. The official page lists 19 teams, 478 valid leaderboard submissions and 30 final submissions. Over the 8 datasets that could be read, the paper's own detector averages 0.657 and would place 16th on a recomputed ranking; the extended contrast gets 0.705 and 11th; a supervised probe trained on RepE at the same block gets 0.784 and 5th. The white-box winner, GMU-Veritaserum, reaches 0.8511 by having the suspect model judge its own transcript, while the white-box baseline Did You Lie probe scores 0.6611.
At the stricter cutoff that flags 1% of honest answers, none of the six detectors evaluated catches half the lies on any model; the top one reaches 0.47. The paper therefore recommends reporting detection at a fixed threshold alongside ranking rather than AUROC alone.
Another team hits the same wall
The mechanism-level cross-check comes from a different group. Cooney, Africa and Irving (https://arxiv.org/abs/2606.12618) build 13 reasoning model organisms whose beliefs are verified in chain-of-thought, so the model genuinely believes what it denies. Across 31 open-weight models from 2B to 1T parameters, all four of their detectors scale positively with model capability on prompted lying. On trained organisms, though, every activation-based and logprob-based detector drops sharply, and only the chain-of-thought judge stays strong, at 0.82 balanced accuracy.
Two papers, different paths, the same wall: there is signal in the activations, but it tracks what the model just said more closely than whether it is deceiving you.
So what
The value of this paper is not a working lie detector but the dismantling of a claim people wanted to believe: cheap, internal, label-free detectors sound like they sidestep every inconvenience of supervised learning. What they measure turns out to be substantially just wording. And the line worth remembering is the paper's own reminder: a detector that ranks lies well may flag none of them.
Two things I would watch next: whether the knowledge direction survives a change of wording, which is an open question the paper lists, and whether any detector can catch the lie before the model admits it. The competition winner at 0.8511 offers an unflattering answer: make the model speak first, then read it.