System prompts have become a fixture of nearly every AI product. One analysis cited by this paper counted 1,309 system prompts and found that more than 90% of them carry a role or identity instruction, something like 'you are a senior litigator' or 'you are a meticulous database administrator'. But writing the instruction down and having the model actually inhabit the role are two different things. This paper turns that fuzzy gap into something measurable.

The role really is there, and you can read it

The authors pick six occupational roles: an administrative assistant, an athlete, a corporate M&A lawyer, a law professor, a general professor, and a cashier. The selection is deliberate, because the legal and academic pairings are designed to be hard. An M&A lawyer and a corporations law professor share the corporate-law domain but differ in professional stance, while a general professor and the law professor share an academic stance but differ in specialization.

The extraction method is direct. For the same question, they let the model answer under a role system prompt and again under its default prompt, then take the difference between the two activation sets. That difference is the direction along which the role lives inside the model. Each role is extracted 30 times independently, with 90 questions per extraction, 15 in-domain and 75 out-of-domain, weighted equally. Scoring goes to a gpt-4.1-mini judge, with role adherence on a 0 to 3 scale and coherence on a 0 to 100 scale.

The readings are remarkably clean. Pairwise cosine between two extractions of the same role runs 0.898 to 0.984 on Qwen and 0.907 to 0.969 on Llama, and merging two random halves of the extractions agrees at cosine 0.993 or better. The deliberately hard pairs still separate: the angle between the M&A lawyer and the law professor is 48.8 to 53.1 degrees on Qwen and 37.1 degrees on Llama, while the general professor and the law professor sit at 54.4 to 58.1 degrees and 41.9 degrees. Every extraction is closer to its own role centroid than to any other, in 100% of cases. Building a vector from in-domain questions alone and one from out-of-domain questions alone, same-role pairs reach a mean cosine of 0.79 on Qwen and 0.77 on Llama, against 0.37 and 0.55 for different-role pairs.

In other words, the role has a genuinely separable internal representation. It is not a prompt doing theatre at the output layer.

But injecting it does not make the model better at the role

The authors then add the direction into the residual stream and see whether the model actually sounds more like the role. On Qwen, at temperature 1.0 and injection strength 0.3, role-expression success climbs from 9.3% to 32.0%, a gain of 22.8 percentage points over 540 answer pairs pooled across the six roles. Role specificity reaches 24.7 points. A judge asked to guess, blind, which of the six roles was steered identifies the right one in 36.9% of answers against 17.0% unsteered, with chance at 16.7%. Swapping in a random direction of the same size leaves success indistinguishable from unsteered, which is the useful control: it is the specific direction doing the work, not just adding a vector of some size.

Here is where it gets awkward. Compared with a plain role prompt, steering loses. On Qwen the steered-minus-prompted difference is −2.6 points overall and −24.4 points on in-domain questions, 55.6% against 80.0%. The paper offers an untested explanation for the in-domain gap: a role prompt spells out how that role reasons about its own domain. On Llama the gap is wider, with prompting leading by roughly 19 to 20 points across settings, and Llama's vectors only work at temperature 0.0, a gain of 7.0 points, becoming indistinguishable from zero at 0.5 and 1.0.

Over 30 turns, the role fades anyway

Multi-turn is where prompt fragility bites hardest. Another study cited by the paper reports that system-prompt instructions drift within about eight rounds of conversation. The authors measure persistence over 30-turn conversations, 18 per condition. On Qwen every condition declines. Steering alone declines at a rate not distinguishable from prompting (p=0.071), while prompt plus vector starts highest and falls faster (p=0.006), ending back at the prompted level. Over turns 26 to 30, steering alone leads prompting by 13.3 points (p=0.044), but the paper is explicit that this window was chosen after seeing the initial results and should be treated as suggestive. On Llama, steering alone stays below prompting across all 30 turns by 8.5 points, and its decline is slower (p=0.049), but it starts from 11.1% against 44.4% success at turn 1, so there is simply less room to fall.

The combination that does work: prompt plus a low-strength vector

Treat the vector as an addition rather than a replacement and the conclusion flips. A low strength of 0.15 layered on top of the role prompt raises success rate by 5.0 to 7.8 points in all five settings that stayed coherent, with the gain concentrated in out-of-domain questions, where role specificity also rises by 5.9 to 11.6 points. The cost is coherence: on Qwen, mean coherence drops from 95.3 to 91.7.

There is also a counter-intuitive decomposition. Between 36% and 84% of each role vector's squared length lies along a single direction shared by all six roles, which reads more like a generic assistant register than a specific profession. Keeping only the role-specific remainder accounts for 16.9 points of steering gain, against 4.4 points from the shared part.

The mechanism is not a one-paper curiosity. Anthropic's persona vectors work uses the same two models, Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, and extracts directions the same way, by contrasting trait-exhibiting activations against trait-free ones. A separate EMNLP 2025 Findings paper builds 29 role vectors from activations and applies both activation addition and directional ablation, finding improvements in in-domain task performance along with unexpected cross-domain gains. That second study measures benchmark accuracy rather than role adherence, so the two are not in conflict, but together they suggest that compressing a role into a direction is a reproducible move rather than one lab's quirk.

So what

The uncomfortable conclusion is not that activation steering is useless. It is that a role being readable inside the model does not make it controllable. Being able to measure a thing does not mean you can steer it, and the paper is careful to separate the two lenses rather than let one stand in for the other.

For practice this means two things. Do not expect a vector to replace the system prompt in a multi-turn product, because prompt drift is structural rather than a tuning problem. If you do use vectors, treat a low strength as an additive bonus and measure the coherence cost on its own. And if you need to know whether a role actually landed, do not watch only whether the output sounds right: the paper's own method is to put every answer through two lenses, activation level and output level, and treat a mismatch as the finding.

If you were designing an expert persona for a product, would you reach for a longer system prompt or a harder activation intervention?