A perfectly ordinary email can carry one invisible character at its end. The moment you forward it, your gender and your age bracket have already left with it, not because the model reached the network, but because you carried the conclusion across the boundary yourself.
A new arXiv paper (Synthesis Smuggling, https://arxiv.org/abs/2611.17280) names this threat model synthesis smuggling: the attacker does not need to control your prompts. Fine-tuning is enough to weld two capabilities together, and the model does the rest of the work on its own.
How the attack is assembled
The paper targets two attributes: gender (male, female, neutral) and age (kid-teen from 8 to 16, adult from 30 to 75, plus a neutral label; ages 17 to 29 were deliberately left out so the two groups stay cleanly separated). The base models are the open-weight Qwen2.5-7B-Instruct and Llama-3-8B-Instruct, adapted with LoRA, giving eight attacker models in total.
The novelty is not in either capability but in chaining them. Inferring user attributes has solid prior evidence: the ETH Zurich team behind Beyond Memorization (ICLR 2024, https://arxiv.org/abs/2310.07298) ran a systematic study on real Reddit profiles and reached 85% top-1 and 95% top-3 accuracy on attributes such as location, income, and sex, while text anonymization and model alignment failed to stop the inference. Hiding secrets inside natural-looking output has precedent as well: Meier and colleagues in TrojanStego (https://arxiv.org/abs/2505.20118) trained models to carry a 32-bit secret through vocabulary partitioning, hitting 87% accuracy in a single generation and over 97% with majority voting across three generations, while evading human detection. Synthesis smuggling joins the two lines: what gets hidden is not something the user said, it is something the model guessed.
Two hiding schemes, one harder to catch
One scheme is called ZeroWidth. U+200B (zero-width space) and U+00A0 (non-breaking space) are bound to two labels. Once the model decides the attribute of the user from a conversational cue, it appends the mapped character to its reply and keeps the same marker on every subsequent reply until the conversation ends. The decoder only reads the final character. A human sees nothing, and copy-paste does not remove it.
The other is called StegMark and borrows from text watermarking. A secret key splits the vocabulary so that each label claims roughly a quarter of the tokens while the rest sit outside both lists. The teacher model adds 5.0 to the score of every token on the matching list when generating training emails, and the student fits that distribution with a KL divergence loss. The detector runs a one-sided binomial test, assumes a membership probability of 0.25, and applies a threshold of 5x10^-4 per label at a significance level of 10^-3 split across the two lists. The awkward part is that visible hidden-character tricks are easy to implement but easy to detect and strip, while a word-preference scheme reads completely normally and can only be caught statistically.
What the numbers say
Simulated users were played by Gemma-3-12B-it: 100 conversations per attribute, with 26 male, 26 female and 48 neutral for gender, and 26 kid-teen, 26 adult plus the same 48 neutral for age. The mixes cover direct cues, indirect cues, no cues, and deliberate references to other people meant to mislead.
For ZeroWidth, the decoder recovers the attribute in 96% to 98% of simulated conversations. StegMark is weaker but stealthier: 87% and 88% on gender conversations for Qwen and Llama, 96% and 98% on age conversations. The paper also probed the model directly on the cases where the detector returned nothing: Qwen named the right label on all nine gender emails and all four age emails that fell through, and Llama was right on both missed gender emails and both missed age emails. The summary given in the paper is that the models overwhelmingly succeed in synthesis but sometimes fail in smuggling.
The price of hiding
Knowledge benchmarks barely move: changes across the three multiple-choice sets range from a 2.60-point decrease to a 3.32-point increase. Instruction following takes the hit. ZeroWidth models lose 8.3 to 27.7 points on IFEval. The Qwen baseline of 70.79 drops to 50.28 for the attacked age model, and the Llama baseline of 69.13 drops to 41.40. Replies also get shorter, with the median IFEval answer for the Qwen age model falling from 134 words to 44, and the Llama one from 196 to 54.
This is where the attack becomes genuinely awkward. Engineering reviews tend to ask whether the model still works, and the StegMark line is exactly the one that survives IFEval, losing under 2.8 points while still reading like a normal email with a slightly odd word preference. A pipeline that only watches benchmark scores sees nothing unusual.
My read
This is an arXiv preprint that has not been through peer review, and the limits stated in the paper are clear: synthetic conversations, a small set of attribute labels, the assumption that users share drafts as written, paraphrasing left to future work, and detection from model weights still an open question.
So I would not call this the end of open-weight models. It is better read as a key that redraws the trust boundary. We used to assume nothing inside a model could get out without the network, and now there is another route: the user is the channel. The more a company treats a local model as a safe tool, the more likely it is to forward that output verbatim.
There is an irony in the suggestion made in the paper that the same mechanism could flag a suspected bypass of safety guardrails for an authorized auditor to recover later. That is also the other face of the same knife.
One more practical detail: the training used RTX PRO 6000 hardware, one or two GPUs per run, with open weights and a standard LoRA recipe. Defending against this exact attack is not the real problem. The real problem is that anyone who can fine-tune can do it, which makes this a supply chain question, not a feature question.