Jev, Microsoft-Decision-1, and the OpenAI Decisions API have been released at a pace since September that the industry is now asking a sharp question: how do these decision models actually compare to general LLMs? Existing benchmarks (JevBench and ReflexBench) each contain only a few hundred questions, so rankings flip whenever a handful of tasks are swapped. System1-Arena is built to give the field one shared scoreboard.
92k questions plus 138 closed loops, the first truly fair benchmark
System1-Arena represents every task as a state plus typed questions: boolean, multiple choice (2 to 255 options), or ordered scores (2 to 10 levels). A model returns a probability distribution per question, no text generation. The static portion alone contains 92,125 questions across 105 tasks from 268 public sources in 16 domains. The agentic portion adds 138 closed-loop environments from 18 families, covering web shopping, SWE-bench coding control, robotics, games, and driving. In real-time mode, the world keeps evolving during the model call. Thirty models share the same evaluation: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 decision models that consume the typed interface directly.
Static decisions: large decision models nearly match fast closed LLMs
On non-agentic tasks, Decision-2.0-Vega-27B (Jev family) and Jev reach 73.0% and 72.6% overall accuracy, beating GPT-6 Luna at 71.2% and GPT-5.6 Luna at 70.0%. Both sit within 0.9 to 1.3 points of Gemma-4-31B (73.9%), the strongest open LLM. On Brier score, the two decision models post 0.343 and 0.349. Only four models do better, all closed LLMs, each costing 20 to 105 times GPT-6 Luna. Decision-2.0-Vega-27B runs about 0.05 USD per 1,000 requests; Microsoft-Decision-1 drops to 0.038 USD. The most accurate model, Claude Opus 5.5 at 81.1%, costs 10.38 USD per 1,000 requests.
Reasoning lifts accuracy but breaks probability calibration
The paper gives four open LLMs a budget from 0 to 2,048 reasoning tokens. With zero thinking tokens, all four underperform Jev (59.3% to 70.2% versus Jev 72.6%). Small budgets sometimes make things worse: DiffusionGemma falls from 69.7% to 56.3% at 128 tokens. At 2,048 tokens, three of the four surpass Jev; Qwen3.8-27B climbs from 70.2% to 75.6%. The same model, however, sees its Brier score worsen from 0.391 to 0.430, and mean confidence jumps from 81.4% to 97.4%. The model concentrates probability mass on a single option, including when that option is wrong, and the Brier penalty grows. The trade is real: reasoning can buy accuracy, but it spends calibration.
Closed loops: large models still win, small decision models are out of their depth
The closed-loop leaderboard looks completely different. Nine models cluster between 67.4% and 71.3% step accuracy, yet their closed-loop success ranges from 5.7% to 32.9%. Decision-2.0-Vega-27B, Jev, and Qwen3.5-9B all post 67.8% step accuracy, but their loop success is 26.5%, 17.4%, and 5.7%, respectively. Errors compound over an episode, and a static step does not predict a closed-loop rollout. Gemma-4-31B leads closed loops at 32.9%. The four open LLMs below 10B parameters all sit under 6%. Jev at 17.4% beats six of the open LLMs but trails both Luna models (22.9% and 18.9%).
What this means
Decision models are not "cheaper LLMs" and not "smaller and thriftier". They are a distinct class built for structured probability output, with strengths in short decisions, low latency, and calibrated confidence, and weaknesses in long-horizon planning. System1-Arena makes the boundary quantitative: for routing, content moderation, and ticket triage where stable confidence matters, Decision-2.0-Vega-27B, Jev, and Microsoft-Decision-1 are already enough, and their calibration beats GPT-6 Luna. For agent loops, calling on System 1 is only a temporary shortcut before System 2 has to take back over.
This matches the framework Kahneman laid out in Thinking, Fast and Slow forty years ago: System 1 handles familiar, repeatable judgments; System 2 handles unfamiliar ones that need planning. System1-Arena gives that intuition its first 30-model, 90,000-question dataset. The real question for agentic AI is not "let the model decide for you" but "let the model decide what should be left to System 1 and what must stay with System 2".