Crafting an iron pickaxe from scratch in Minecraft is a multi-stage pipeline — mine, smelt, craft — and until now the AIs that managed it either saw the world through environment APIs or had their hands scripted. HiSTEVE 1.0, submitted to arXiv on October 22, offers a third answer: first-person pixels only, native keyboard-and-mouse only, the whole chain run the way a human player would. The Hugging Face paper page badges the team as Tencent.
The old gap between two agent camps
Minecraft agent research has long split in two. The VPT and STEVE-1 lineage has native visuomotor control but no long-horizon autonomy; Voyager-style planners run high-level plans through environment APIs and predefined skills, and go limp without privileged interfaces. HiSTEVE stitches the two together — it reads no internal game state and calls no environment API, pairing hierarchical planning with native execution in what the paper calls a capability-aware agent harness.
The hierarchy: planner, detector, executor
At the orchestration level, a structured memory tracks task progress while GPT-4o acts as the planner, picking the next atomic subtask from within the VLA's capabilities. At the control level, a single VLA initialized from Qwen2.5-VL-7B takes vision-only input and emits keyboard-and-mouse actions at 20 Hz across all subtasks.
The hardest part of any hierarchical system is knowing when a subtask is done. The team trained a policy-coupled task-completion detector: a Qwen2.5-VL-3B backbone sharing the VLA's image encoder, reading historical frames sampled at 800/600/400/200 ms. On a class-balanced validation split of 4,674 VPT-trajectory samples, this detector — listed at 1.4B in the paper's table — reaches 0.863 in both accuracy and F1, ahead of prompted GPT-5.5 at 0.776; Qwen3.5-35B manages only 0.042 recall on the incomplete class. A purpose-built small model beating a general-purpose large one on a narrow job — a clean example.
Two engineering details easy to miss
When GUI and non-GUI interactions share every parameter, their optimization signals fight. The fixes: regime-specific RMSNorm, giving each interaction regime its own per-layer scale vector at a parameter cost of roughly two-times-layers-times-hidden-dim — negligible; and a separate token embedding for mouse movement. Training proceeds in three stages: world-knowledge injection, imitation learning, then reinforcement learning with environment feedback.
The scoreboard, and the cold water
In the paper's own comparison, HiSTEVE scores 85.0% strict pass@1 on MineStudio Bench (20 atomic tasks, three trials each) against 65.0% for the strongest baseline, OpenHA Text/CoA. On OpenHA it averages 37.4% versus 36.8%, with the gain concentrated in the Embodied family (37.0% up to 46.0%). On the long-horizon LongBench (14 goal-only tasks, 49 hidden diagnostic nodes), the Resource & Crafting family reaches 42.9%, a 28.6-point jump over the baseline.
But the paper is blunt about the weak flanks: GUI and Combat families sit slightly below the baseline — failed GUI episodes reach the right interface, then break on multi-step item placement, intermediate-state correction, or result retrieval; Recovery and Open-ended tasks remain hard. And since the planner is closed-source GPT-4o, a stretch of the system's "autonomy" lives in someone else's API.
Why games deserve a serious look
Games are the cheapest stress-test arena for embodied intelligence: open rules, instant feedback, zero-cost failure. More to the point, the human-interface route — no privileged API, just screen and keyboard — is the mirror problem of real-world GUI agents: making an AI operate an ordinary computer and making it play an open-world game test the same capability. When this route matures, the first thing redefined will not be games, but "work that happens at a computer" itself (full paper: arXiv:2610.31162).