Recursive self-improvement is widely seen as the next step for agent technology, but any team that has actually run a discovery loop knows one component is still hand-written: the exploration policy — where to branch, what to run in parallel, when to cut off a dead line. A team from Google and Google DeepMind, together with the University of Maryland and the University of Virginia, takes aim at this bottleneck in the Dream-RSI paper released on September 14 (arXiv:2609.14858, project page).
The exploration dilemma
The paper's analysis is blunt: a fixed strategy cannot learn from accumulated experience, so it keeps paying for directions that have already failed. Optimizing the policy online, meanwhile, hits two walls at once — meta-level feedback is delayed and expensive, because judging an exploration policy means watching it steer an entire discovery run to the end; and the policy space is vast, so most candidates you would need to try are bad ones, each costing a full rollout to find out.
History is already a simulator
Dream-RSI's key insight: you have already paid for that feedback. A finished discovery run is not just text to re-read — it is a structured tree of every exploration decision the agent made, each carrying the execution outcome it actually produced. And an exploration policy does exactly one thing: given what it has seen, choose which attempt to continue next. So an alternative policy never has to re-run anything. It walks the same recorded tree in a different order, and every outcome it asks for is already on disk. The paper claims thousands of candidate policies can be screened at zero execution cost, with a real rollout spent only on the winner.
Notably, this simulator is not a learned world model — it is exact. Over the search space that was realized, nothing is predicted, because the simulator is that search space. The flip side is just as sharp: a policy can only be dreamt where history actually went. That is precisely why this has to be a loop — every online deployment records a new tree, so what the agent owns is not one world but a growing pool of them; a policy dreamt across more worlds beats one tuned to the luck of a single run.
There is also an elegant design detail: in the offline phase, a policy-development agent writes successive revisions of the policy code, each scored by replay over the whole history. Because the currently deployed policy is itself in the candidate set, the winner is never worse than the incumbent.
The numbers as reported
Across eight discovery tasks spanning algorithm engineering, mathematical optimization, and GPU kernel engineering, the paper reports: on GPU kernel task VGG16, comparable performance with 2.43x fewer generations (1.79x on LayerNorm), and a 2.09x higher score on ConvDiv under comparable budgets. On the Lasso regularization path task, Dream-RSI with Gemini-3.1-Pro spent just 317 discovery-agent calls versus 51,200 for the comparison system SimpleTES — a 162x reduction in calls, as the paper frames it. On mathematical optimization, SimpleTES holds the best single number, but at a cost of 51,200 generations against fewer than 1,000 here. The paper also notes roughly 1.7x fewer discovery-agent calls versus fixed exploration.
More counterintuitive is the analysis section: semantic guidance — abstracting prior trajectories into high-level insights and injecting them into the prompt — consistently underperformed its unguided counterpart under equal budgets. Strong priors over-constrain the search space and suppress exploration diversity. The learned policy is not monotonically greedier either: evaluated attempts per round first fell from 110 to 50, then rose back to around 90 after entering a plateau — and those widenings lined up with the next jumps in round-best score.
So what
Most agent frameworks read history in one of two ways: static text to stuff into the context, or training data to fine-tune weights on. Dream-RSI offers a third reading — history is an environment, a simulator already built and paid for but left idle. The perspective extends beyond discovery tasks: any long-horizon agent that leaves complete execution traces theoretically sits on the same replay asset.
To be honest about the caveats: every number above is self-reported in the paper, the controlled baseline Recursive Fixed Exploration was built by the team itself, and the SimpleTES figures are their reproduction — independent verification will take time. The method also can only dream where history has already gone, so the world pool is small during cold start, and the value only materializes once the loop is spinning. But even at half strength, turning exploration policies from hand-written scripts into evolvable code and pushing meta-level evaluation cost toward zero is a reminder for every team building agent infrastructure: those trajectory logs on your disk may be worth far more than you think.
One line to remember: history is not just context — it is the simulator you already paid for and never really used.