UK fintech firm Saturn released its September "Artificial Authority" report, putting 18 mainstream AI models through a gauntlet of real personal-finance questions. The verdict isn't flattering: 57% of answers contained errors on average. On multi-step, calculation-heavy questions, the error rate jumped to 88%; the hardest items saw some models fail 99% of the time. Even the best performer — Anthropic's Claude Opus 5 in reasoning mode — landed at only 61% accuracy. Pick the top of the leaderboard and you still get a wrong answer roughly one in three times.
What the test looked like
The benchmark covers free and paid tiers from ChatGPT, Gemini, Claude, Copilot and Grok — 18 models in total — against 121 money-related questions, repeated up to five times each, for over 10,000 answers in aggregate. Questions fall into three difficulty buckets: easy (definitions), medium (single calculation), and hard (multi-step computation combined with tax-rule interactions). Grading uses a "paraplanner must-pass checklist" combined with an LLM-as-judge; a response only counts as a pass if it hits every required point, otherwise it is marked a failure.
Three failure modes dominate the report: arithmetic errors (especially around tax rates and pension thresholds), omissions of upcoming tax-policy changes, and rules that were simply invented out of thin air. The FT's coverage (https://archive.is/WD6Vy, original URL https://www.ft.com/content/c0cd359d-df84-4208-a789-ffa864b43666) cites two concrete examples. Claude Haiku 4.5 miscalculated a pension-tax scenario that could have exposed the saver to a £17,500 HMRC charge. In a separate case, Claude "invented" a rule claiming a UK graduate could stop student-loan repayments simply by moving abroad.
Reactions from regulators and industry
Saturn CEO Amal Jolly told the FT plainly that "millions of people are trusting AI models for money advice, but they are getting wrong answers that can lose them money." The report dropped alongside a UK Financial Conduct Authority finding that one in five UK adults is open to letting AI make financial decisions for them, with the willingness skewing younger. Jolly used the moment to push the FCA to bring AI financial advice inside the regulatory perimeter — currently it sits outside, so consumers have neither compensation rights nor a clear path to redress. Sarah Coles, head of personal finance at AJ Bell, gave a more measured response: AI is fine for budgeting and initial research, but anything involving tax, debt prioritisation, or retirement planning "is exactly why professional advice still matters."
Methodology pushback
The report has not gone unchallenged. The sharpest critique hits the methodology: the grader is itself an LLM, judging the same kind of object it is supposed to be judging — referee and player are the same team. A large slice of the "failures" (~37% by one commenter count) are flagged for incompleteness rather than factual error — missing a required warning, missing a compliance step — which is a softer failure mode than the headline implies. Saturn's day job is selling AI compliance tools to 750 financial-advisory firms, so the commercial angle is visible. The full report sits behind an email gate, the question list and prompts aren't public, and third parties can't reproduce the run. One FT commenter cut straight to the point: "A company selling the answer to the problem publishes research concluding the problem is real — go figure."
What this actually means
Setting the methodology debate aside, the headline numbers still deserve attention. An 88% error rate on complex items is not marketing copy; it means that in multi-step, cross-rule scenarios, the financial-QA capability of mainstream LLMs is well below what a casual user would assume. Combine that with the model's confident tone and its polite, fluent delivery of errors, and a non-expert has almost no way to spot a wrong answer on their own.
A few practical rules for anyone using AI for money questions: when the prompt touches numbers, thresholds or jurisdiction, force the model to show the calculation step by step and to label every assumption. For any question about a policy that "takes effect soon," go to the primary government source and compare. Treat the model as a draft generator and a sanity-check table — not as an adviser. When the model says "you can…", assume it's wrong until you've proven otherwise.
Saturn's biggest contribution is not a public shaming of AI. It's a temperature check on the "AI replaces the professional" narrative. In domains with high expertise floors and long liability tails, "usually works" is not enough; "rarely fails" is the bar. Next time an AI finance tool pats itself on the back, ask it to run this 121-question battery first — and see how confident it still sounds.