In mid-September, Jev turned the "decision model" into a product category: no prose, just one forward pass that outputs a probability for every option. Two months later the field holds 28 models worth comparing side by side — closed-source Jev 1.14.0, Microsoft Decision-1 and GPT-6 Luna, alongside open-source WeMM-JEV, KnowLine, Perplexity Decider and Clef. On November 10, a team from CUHK and HKUST posted a new exam to arXiv, MMDM (arXiv:2611.14309), which puts all 28 models on one scoreboard — and the results no longer match the launch-day claims.
An exam built against memorization
MMDM's 5,448 questions span 16 task categories: world knowledge, reading comprehension, deductive and quantitative reasoning, agent state tracking, tool and skill selection, preference judgment, and rubric scoring. There are three answer types: 4,112 hard single-select questions (one correct option), 633 hard multi-select questions (the complete valid set is required), and 703 soft-label questions (the full probability distribution must be matched). Option sets run from 2 to 1,024 options, with 94 questions exceeding 255; measured with the Qwen3.5-4B tokenizer, complete inputs have a median of 827 tokens and a maximum of 64,817.
Every question is freshly constructed: existing public datasets are rewritten while keeping their original answers, new questions are recomposed from source data, and entirely new tasks are written — then screened by a five-model majority vote plus independent human expert review, with an option-only probe (options shown, state and query hidden) to catch shortcut answers. Distractors are not thrown together either: they are designed around identifiable failure modes such as applying an expired rule, confusing similar entities, or overlooking an action's prerequisite. The motivation is stated up front: existing decision-model evaluations reuse public benchmarks heavily, making test-set contamination a first-order concern; and among the prior decision benchmarks compared in the paper, the multi-select column is empty across the board.
The scoreboard: knowledge easy, joint reasoning not
The best single-select score belongs to open-source WeMM-JEV 27B at 76.00; the best closed-source model, Liquid-d1-Ultra, reaches 73.32; Microsoft Decision-1 sits at 61.36, fourth among closed-source entries. Its launch framing told a different story — according to The Decoder, Microsoft reported 83.5% average accuracy and first place in its own 36-benchmark comparison of roughly 150,000 questions (source). On an exam the models have not seen, the score is the score.
Broken out by category it gets worse. Perplexity Decider-V1.2 scores 78.75 on world knowledge but 53.95 on option logic and 47.95 on constraint planning — it reads the world fine but cannot work out dependencies between options. Microsoft Decision-1 manages just 13.5 on constraint planning, ahead of only five models at 4B or smaller. Multi-select is a collective failure: the best accuracy across all 28 models is 36.81, again WeMM-JEV; on SB-F, which measures whether a model recovers the complete valid set and distributes probability over it sensibly, not a single model reaches 0.20.
Picking the right answer is not the same as usable probabilities
The real selling point of decision models is being used as a probability source, and the paper isolates that layer. WeMM-JEV and KnowLine 31B differ by 0.93 points in single-select accuracy (76.00 vs 75.07), yet KnowLine's Soft Brier of 0.1233 is about half of WeMM-JEV's 0.2327, and its Score nMAE of 0.0624 is less than a third of 0.2203. Calibration also stratifies: among five representative models, Microsoft Decision-1 has the largest ECE and stays overconfident through the mid-to-high range, while OpenJev's top confidence bin averages 0.97 confidence against 0.87 accuracy. Select on "how often it picks correctly" alone, and any downstream pipeline that thresholds on the probabilities is walking into a trap.
Pick by category, not by total score
Category champions are often not the overall champions: the best evidence-validation score belongs to Quyet-1.0-Large at 89.1, and the best emotion recognition to Mapika Decider 4B at 64.0. Routing skill does not transfer either: GPT-6 Luna Decisions beats Jev (65.3 vs 63.4) and Microsoft (62.6) on intent routing, yet scores only 16.8 on workflow decisions, far below their 52.2 and 43.5. Models with near-identical totals can have completely disjoint strengths — without checking which column your workload lives in, model selection is close to a lottery.
So what
This wave of decision models is not about being smarter; it is about being fast, cheap, and pluggable into pipelines. But MMDM's numbers say probability quality has not kept pace with the shipping speed: multi-select recovery, constraint awareness, and option interdependency are unsolved across the entire field. The dataset is open on Hugging Face, with code and the full leaderboard on the project page. Before deploying, map your own workload onto those 16 columns and choose accordingly — that beats chasing the total-score leaderboard by a wide margin.