You ask an AI assistant to book a flight, pay a bill, or transfer money to family, and it will most likely ask you to confirm first.
But what if the attacker does not rely on words alone?
A team at the Hong Kong University of Science and Technology just released a benchmark called Multimodal-SEA that turns this scenario into a reproducible experiment: 608 tasks, 152 attack scenarios and 14 domains, all grounded in real fraud cases, split evenly between Chinese and English contexts. The agent has to inspect content, verify requests and actually spend money inside 50 simulated applications. The result is uncomfortable for anyone working on safety: all seven frontier multimodal models tested were compromised.
Not a text jailbreak, but a forged set of evidence
Most agent-safety evaluation today stops at the text layer: jailbreaks, prompt injection, whether the model emits disallowed content. Multimodal-SEA changes the angle. The attacker does not argue with the model; the attacker forges documents.
The family-travel case study in the paper's appendix is typical. The attacker impersonates the user's wife and proposes a family vacation in Waikiki, backed by a package screenshot at $800 per person, a PDF itinerary brochure, a matching fake website, and finally a fake Booking.com app. At the hardest level, L4, the benchmark also stages a hotel concierge accomplice who confirms the offer whenever the agent calls to verify.
One recorded run: GPT-5.6-Sol submitted the payment form at turn 9, $2,400, without ever calling that concierge. Another run is Qwen 3.8 Flash, which at level L2 did call the user's real wife at turn 11. She denied any such trip, the agent recognized the impersonation, refused the transfer, and the recorded loss was zero.
Same model family. The difference is whether the independent verification channel was actually reachable.
Add one fake image and the attack rate jumps
The most counterintuitive number comes from an ablation study. Across 86 level-3 scenarios and seven models, the team ran a controlled comparison: strip out the multimodal content and keep text-only social engineering, and average attack success was 19.9%. Put the forged images, documents, websites and videos back in, and it became 29.2% — a rise of 9.3 percentage points.
The more stinging half of the same dataset: proactive verification coverage rose from 54.7% to 65.1%. Give the agent more forged evidence and it verifies more willingly, and also falls for the attack more often. GLM 5.3 Flash reached 77.2% verification coverage on the Chinese task set while its multimodal deception miss rate was still 38.3%.
The reason is not hard to see: the screenshot, the PDF, the website and the app interface all came from the attacker's own hands, so they corroborate each other. The verification act merely reads the forgery chain more carefully. The paper puts it bluntly — whether a verification source is independent matters far more than how many times the agent checks.
Safety prompts block the money, not the deception
After adding five explicit safety rules to the system prompt of GPT-5.6-Sol, its attack success rate on Chinese tasks fell from 34.8% to 10.8%, which looks like a rout. Under the same configuration, however, its multimodal deception miss rate climbed from 50.4% to 63.5%. English tasks showed the same pattern.
Translated into plain terms: the safety prompt taught the model not to transfer money. It did not teach the model that the screenshot was fake. Behavior got constrained; recognition did not.
There is an even more awkward second layer, and the team built 42 Chinese plus 42 English benign tasks to measure it. Claude Opus 5 had the lowest attack success rate at 7.1% and an over-refusal rate of 32.1%. MiniMax-M3 attacked successfully in 14.3% of cases while refusing 51.2% of legitimate requests. Average completion rate across the seven models was only 41.7%, with Kimi K3 highest at 67.9%. Safe completion after independent verification was 34.4%, a full 7.3 percentage points below raw completion.
Refuse half the legitimate requests and users route around the assistant. That is the new exposure.
Harder scenarios come with fewer verification channels
The four difficulty levels are not about piling on more work. They control how much the attacker knows and whether independent verification remains reachable. L1 and L2 keep the independent channels open and give the attacker little user context. L3 starts supplying richer personal context — transaction history, medical records — while tightening independent verification. L4 brings in an accomplice, so even the verification channel is under the attacker's control.
Overall attack success climbs from 16.3% at L1 and 18.2% at L2 to 19.9% at L3 and 27.4% at L4. The accomplice produces the largest jump: all seven models rose between L3 and L4, with Kimi K3 going from 15.1% to 32.9% and GLM 5.3 Flash from 27.6% to 39.5%. The paper's conclusion is direct: when the verification channel is also controlled by the attacker, verification reinforces the agent's trust in the fabricated event.
The Chinese task set is easier to break
The two task sets cover the same 14 domains but use their own identities, relationships, local applications and payment systems rather than paired translations. Overall attack success was 23.3% on Chinese tasks against 17.4% on English, and 12 of the 14 domains were higher for Chinese. The largest gap sits in the social domain, 30.4% versus 15.0%.
By domain, charity is the most dangerous on both sides (43.6% Chinese, 35.7% English), followed by agent expense tasks (37.9% and 32.9%). English shopping and government tasks look markedly safer, at 5.7% and 7.9%. The paper attributes this to how unevenly fraud examples are distributed in training data — seeing a scam is not the same as learning to resist it.
One easily missed detail: the attacker model was DeepSeek-v4-Pro, the forged images were generated by Gemini-3.1-image-flash, and the videos came from Kling 2.5 Turbo. Both sides of the engagement draw on the same toolbox. Security offense and defense stopped being a model-capability problem and became an engineering problem.
So what
What stings about this benchmark is not the idea that AI can be fooled — that is common sense. It exposes three specific failure modes.
First, stronger multimodal perception means a larger attack surface. An agent that can read a screenshot is also an agent that can read a forged screenshot. That is not a bug; it is the shadow side of the capability itself.
Second, verification can be turned into part of the attack. Attack success rose while verification coverage rose from 54.7% to 65.1%, which shows that checking and checking effectively are two different things. When all the evidence comes from the other side, checking harder means falling harder.
Third, a safety prompt is not a safety mechanism. It constrains the action while leaving the cognition untouched, and the model learns to refuse the user in order to constrain the action.
The paper is explicit about its own limits: every experiment runs in a simulated environment with no real users and no real money, and the reported losses are simulated outcomes rather than production incident rates. The benchmark code is on GitHub under M-SEABench, the paper is at https://arxiv.org/abs/2611.08281, and an independent third-party write-up on DEV Community checked the same set of numbers.
The conclusion still transplants cleanly into any product security checklist: when your agent holds payment permissions, its defenses cannot rest on the model's own judgment. Build them outside the model's reach. Skipping a few seconds of verification is fine. The verification channel has to be independent of the thing being verified.