Computer-use agents (CUAs) are improving fast, but the environments for training them remain a bottleneck: native applications are expensive to build and integrate, which narrows coverage and invites overfitting, while browser-hosted environments offer limited interaction depth. SoftwareGym, posted to arXiv on Oct 8 by a Shanghai Jiao Tong University team, takes an automated route: instead of hand-building environments, it synthesizes software applications along a pipeline and verifies they actually work through real GUI interactions.

Why the old paths dead-end

The paper is blunt about the status quo. Native application environments require substantial construction and integration effort, limiting application coverage and raising the risk of application-specific overfitting. Browser-hosted environments are easier to deploy but cap interaction depth and workflow complexity — you can click through web pages, but you cannot practice real desktop software. Scarce training environments directly throttle the supply of scalable supervision for CUAs.

The pipeline: design, build, evaluate, repair

SoftwareGym turns "making software" into a multi-agent pipeline, with code open-sourced on GitHub. Given a software type (e.g. ChatAPP, PicEditor), the pipeline runs in four steps:

  1. Design — FunctionDesigner writes the requirement spec; StyleDesigner and SubStyleDesigner produce multiple distinct UI style blueprints (Avalonia .axml-style .axaml); IconDesigner generates per-style SVG icon sets with hard validation gates.
  2. Build — Coder implements the app (.NET 8 + Avalonia 11); DevCoder reviews and fixes in place. All design and coding stages run through the Codex CLI.
  3. Evaluate — EvalTaskGenerateAgent creates GUI test cases; GUIEval boots OSWorld Docker containers (real Ubuntu VMs) and has a GUI agent actually operate the app, producing a per-round summary.json.
  4. Repair — rounds v1 to v3, each feeding the previous eval summary and review back to Coder/DevCoder for incremental fixes.

One engineering detail worth noting: SoftwareGym reuses OSWorld's Docker-based VM infrastructure and Ubuntu desktop, validating via screenshots and PyAutoGUI actions, so training conditions stay consistent with downstream evaluation — which helps explain why capabilities learned in synthesized environments transfer.

The numbers: 182 apps and an out-of-domain jump

The paper reports 182 synthesized applications across twelve categories with a mean GUI task pass rate of 91.5%, plus SoftwareGym-Bench, a 120-task evaluation suite spanning the same categories. The key result is transfer: after training on trajectories collected from these environments, the model improves Pass@3 from 41.39% to 56.97% on the strictly out-of-domain OSWorld benchmark and scores 40.0% on SoftwareGym-Bench.

Caveats worth noting

The reproduction bar is nontrivial. The README states the OSWorld VM base image (~35 GB) is not included in the package; you must install the Codex CLI (pinned to @openai/codex 0.156.1), the .NET 8 SDK, and a conda environment; model routing reads environment variables only, and the design/coding stages require an endpoint that supports the OpenAI Responses API. Seed materials for media/file-type software must be self-supplied, and container cleanup is manual. The repository currently sits at 5 commits and 16 stars — a typical paper-first, engineering-just-landed state.

The imaginative part of this route is the loop: synthesize environments, collect trajectories, train stronger agents, then use stronger agents to validate more environments. When environments shift from scarce assets to regenerable material, the data bottleneck for computer-use agents finally starts to loosen. Paper and code: arXiv:2610.12611, GitHub MrKiing-Jin/SoftwareGym.