There's an unglamorous but fundamental constraint in agent research: what a model can learn is bounded by the environment it practices in. The environment needs faithful state, rules and dynamics, and realistic observations at the same time — and hitting both across diverse worlds has been the bottleneck. AgentGarten, released on arXiv on October 8 by MirroS-Lab, offers a clean answer. It hit #1 on Hugging Face's daily paper ranking for October 9 with 141 upvotes.
Architecture: code owns the rules, a renderer owns the pixels
AgentGarten splits a "world" into two layers. The bottom layer is simulators and game engines: they maintain persistent world state and execute program-defined interaction rules. The top layer is a shared neural renderer: every world exports depth maps and surface normals through a common interface, and the renderer generates what the agent actually sees. Rules go to code, visuals go to a neural network.
The renderer is not trained from scratch. The team adapts NVIDIA's Cosmos3-Nano video model in three stages: bidirectional rectified-flow training, autoregressive block-causal training sampled with a KV cache, and finally their proposed Adversarial Forcing, which distills the model into a few-step student. Rollout replay is exact and differentiable, so losses on later blocks update how the student encodes history, while a real-data discriminator polishes visual quality. Inference streams block by block with hand-written Triton kernels and CUDA graphs — no torch.compile.
Results: 4 rounds vs millions
The paper's headline number is stark: agents learn core strategies from just 4 rounds of experience, versus millions of rounds for a conventional reinforcement-learning counterpart. In hide-and-seek, per the authors' note on the Hugging Face page, hiders build shelters by round 4 and seekers use ramps by round 10. After each round, experience is distilled into playbooks that later agents inherit and refine — knowledge accumulates as readable text across generations rather than piling into weights.
Equally important is scalability: new worlds are written as code and rendered through the same interface, so environments can scale in number and difficulty alongside their agents.
What's open-sourced, and what's still owed
What's available now is the renderer's training and inference code under Apache-2.0, with weights on Hugging Face; the repo has collected 143 stars in three days. But two items in the README's TODO remain unchecked: the real-time rendering engine (streaming server and interactive frontend) and — crucially — the code worlds and the agent practice loop itself. In other words, only the renderer half of the system has shipped; the full closed loop behind that 4-round result can't yet be reproduced outside the team.
So treat it as a high-quality component of the "world models as training grounds" route rather than a complete system. Whether code worlds can really scale alongside agents, as the paper envisions, will only be answerable once the practice loop lands. The paper is at arXiv:2610.12374, code on GitHub.