Simulating shoppers has always been the hard part of recommendation systems. Predicting whether someone will click next is not enough — what matters is whether a simulated user holds up across a long shopping journey, and whether the simulation still matches what happened in a live A/B test. On 22 October, Alibaba Group, Leiden University, Renmin University of China and the Institute of Computing Technology at the Chinese Academy of Sciences published a paper on arXiv (UserArena) that turns this into a public test bench.

The dataset: 64,729 real shopping trajectories

The paper says the data was recorded on one of Asia's largest e-commerce platforms, without naming it; Alibaba Group is listed among the author affiliations. The scale: 11,010 users, 64,729 session trajectories, 577,670 actions, 566,102 products, 6 user attributes, with clicks spanning 41 product categories. Of 73,717 item clicks, apparel accounts for 54.90%, and the remaining 45.10% is spread across 40 other categories. The long-horizon portion carves out 59,378 chronologically ordered sessions from 5,768 users, averaging 10.29 sessions and 86.36 actions per user. By comparison, OPeRA has 51 users and 527 sessions at 1.00 average sessions, and USB has 5,222 users and 5,274 sessions at 1.00 — that is what a cross-session set should look like.

The data is not propped up by blockbusters: 92.0% of the 67,267 clicked items appear only once, and the most-clicked 1% of items account for just 3.23% of clicks. The action mix is equally realistic — feed-page scrolling at 38.34%, item clicks at 12.76%, session termination at 11.21%, add-to-cart at 3.90%, purchase at 1.02%. The license is deliberately restrictive: non-commercial research only, prohibiting attempts to re-identify users, linkage with external personal data, and unauthorized redistribution. Interface images are reconstructed in an offline React environment and contain no real user GUI screenshots. Code and data live in the RecGPT-Team/UserArena GitHub repository.

Five evaluation levels, from "parses" to "reproduces production"

Evaluation runs across five levels: environment compliance (L0), local behavioral fidelity (L1), sequential behavioral fidelity (L2), personalized population fidelity (L3), and outcome fidelity (L4). Two modes support them. Replay predicts the logged next action from recorded states, isolating local accuracy; Rollout lets the simulator choose actions and drive states until the session ends, testing whether free-running sessions reproduce real trajectories. L0 runs under both modes, L1 under Replay, and L2 through L4 under Rollout.

OPTD: distilling on the student's own trajectory

The paper identifies the weakness of two standard approaches. SFT learns only from recorded responses, creating a train-test mismatch. GRPO assigns one scalar reward per generated response, which is coarse credit assignment. OPTD relocates the supervision: every token of the student's own rollout is supervised by the full-vocabulary distribution of a frozen teacher conditioned on trajectory-level intent as privileged context. A per-token trust region — picking an interpolation coefficient against a KL budget — pulls that teacher back toward a reference conditioned only on what the student can observe, and a separate grounding term aligns against the recorded real response, with a GT coefficient of 0.1. In ablations, removing any single component degrades performance on at least one metric, and removing the SFT warm-up degrades it most.

The paper's own numbers: a 2B model beating GPT-6

The paper's production A/B validation is the sharpest part. The same pair of ranking policies is run in simulation and in the control and treatment arms of a live A/B test, comparing the ratio of simulated to online ΔPV-CTR. OPTD reaches 0.713× (effect error 0.287), clearly ahead of second-place STA at 0.297×. The other task-specific simulators compress the online effect down to between 0.020× and 0.297×, and the untuned Qwen3.5-2B reaches only 0.004×. Going the other way, GPT-6-Sol at 1.576× and GPT-6-Luna at 2.154× overestimate the online response. The paper's conclusion is that general-purpose model capacity does not automatically translate into behavioral calibration.

In the cross-session table, OPTD takes first place on five of the eight L1-to-L4 metrics and second on two more: DM-JSD 0.0284, trajectory error 0.72, population fidelity PF 0.8929, terminal-state T-JSD 0.0077. Macro-F1 and ΔCTR go to STA. The authors also run a source-blinded Turing-style human evaluation: 200 real-versus-simulated trajectory pairs, three annotators each, with sources anonymized and order randomized, where OPTD posts the lowest human win rate and the highest model win rate.

And yet the leaderboard is empty

One detail deserves its own note: the project page leaderboard states that no verified SOTA scores exist yet, that only complete Replay and Rollout reports are ranked, and that no scores are estimated. The dataset itself is still marked as planned, and the GitHub repository has 3 commits and 3 stars. Anyone wanting to reproduce this will be doing the work themselves.

So what

The real value here may not be "one more benchmark" but the substitution of a falsifiable question for a long-standing matter of taste: instead of asking whether a simulator is accurate, ask whether it reproduces the effect ratio of a live A/B test. Once you frame it that way, a dedicated 2B model beating GPT-6 stops being surprising — simulating a user requires matching behavioral distributions, not absorbing more knowledge. The empty leaderboard, dataset still pending, and non-commercial license all signal that what exists today is closer to an evaluation protocol than a ready-to-use resource.