Most world models stay at the pixel level: frames keep getting prettier, but once the camera performs unconventional motions like roll or pitch, gravity drifts, geometry collapses, and the illusion breaks. A joint team from S-Lab at Nanyang Technological University, the University of Michigan, Beijing Jiaotong University, and ACE Robotics took a different path: instead of patching afterwards, treat physics and geometry as native states of the model (arXiv:2609.04196).
Three Native World States
Puffin-World is a unified multimodal architecture that jointly models three complementary world states: physics (gravity field and latitude), geometry (depth), and appearance (image). Architecturally it combines a geometry-aligned vision encoder, an LLM, a diffusion model, and a lightweight connector, with understanding and generation sharing the same parameters — the same model can interpret camera geometry and synthesize world-consistent observations, without task-specific external geometry modules.
Two designs hold the line: the Omni-Camera representation pairs a gravity-aware absolute perspective field with ray-based relative geometry, using a 9-channel condition for precise control over camera intrinsics, orientation, and trajectory; and a physics propagation strategy carries physical constraints forward along generated trajectories, preserving gravity consistency under extreme rotations and long camera motion.
First on Median Error Across Four Benchmarks
The project page reports that on Stanford2D3D, MegaDepth, TartanAir, and LaMAR, Puffin-World takes 12/12 best median-error results and 33/36 best AUC metrics; roll error reaches 0.26° on LaMAR and vFoV error 1.62° on Stanford2D3D. On the generation side, its self-built Puffin-Cam-Bench shows median up-vector / latitude / gravity errors of 0.84° / 1.26° / 0.79° with the lowest FID; on RealEstate10K it ranks first in both PSNR at 17.22 and LPIPS at 0.318; and on Puffin-Traj-Bench its median roll/pitch errors are 0.80° / 1.10°. For closed-loop applications the team demonstrates mimic world exploration and self-calibrated exploration, where the model reasons about the current physical state and predicts corrective camera actions.
Where 44.5M Camera-Labeled Images Come From
Scaling is the key: the Puffin-16M dataset comprises 15 million vision-language-camera triplets and 1 million diverse camera trajectories. The team also annotated roll/pitch/vFoV for 28 widely used public datasets — roughly 44.5 million camera-grounded images in total — all released through a Hugging Face collection. Code, models, and datasets are all open-sourced across three layers.
The Cold Water
Details hide in the comparison table: on vFoV, AnyCalib, a specialized camera-intrinsic estimator, still beats Puffin-World on LaMAR with a 2.25° median error versus 2.73° — specialized models have not exited the stage on specific metrics. And "12/12 first-place" figures come from the team's own project page against their chosen baseline set, with no independent replication yet; whether every single capability of a unified multi-task architecture withstands pressure from specialized models will take third-party evaluation.
For world-model builders, though, the directional signal is clear: beyond pixel fidelity, modeling gravity and depth as first-class citizens — plus an open-sourced base of 44.5 million annotated images — moves spatial intelligence another step from "can paint" toward "understands physics".