For robots, learning a new manipulation skill is rarely the hard part — the hard part is where demonstration data comes from. Teleoperating a real robot to collect demonstrations blows up any budget, and the long tail of real-world tasks cannot wait. RoboTok (arXiv 2609.03199), from Rice University's RobotPI Lab with NVIDIA, flips the premise: humans have already uploaded the manipulations to video sites; what is missing is a reliable retriever.
Method: compare hand trajectories, not pixels
RoboTok converts internet manipulation videos into 3D hand trajectories expressed in an estimated torso-relative reference frame. Because motion is described relative to the actor's own body, camera viewpoint, scene appearance, and occlusions cancel out, and similar manipulations across videos become directly comparable vectors. A compact embedding space is trained with DTW-based motion-similarity supervision; queries are cosine-similarity search over embeddings, with no pairwise DTW over the corpus — that is what makes internet-scale continual indexing feasible. Even when the actor is out of frame, the model estimates the torso frame from the two hand trajectories alone. The corpus holds 100,000 clips: 90,000 for training, 10,000 held out as evaluation queries.
Retrieval table: 0.3531 versus 0.0071
On the in-domain evaluation set, RoboTok reaches mAP@20 of 0.3531 against 0.0071 for the strongest prior method, STRAP (ICLR 2025) — roughly a 50x gap; MRR@20 is 0.8576 versus 0.0824. On AssemblyHands, an out-of-domain corpus of 831 two-hand assembly clips with sensor-grade 3D hand annotations, mAP@5 is 0.2614, about double STRAP's 0.1330. By DTW cost, RoboTok's retrieved results sit at 1.333, close to perfect retrieval's 1.145.
VTDexManip: hard mode is nearly one-sided
Downstream is the point. Per the paper, demonstrations retrieved by RoboTok give policies the highest success rate on 5 of 6 original VTDexManip tasks and on all 3 modified hard tasks (free 3D hand motion, dense rewards removed). Hard-mode Lever Sliding: RoboTok 79.3% versus 19.5% for the best baseline, HAND (ICRA 2026); hard-mode Faucet Screwing: 44.8% against 6.8%. Policies use only proprioception and fingertip force — no vision.
Cold water: every number is in simulation
The real-world section of the project page reads "coming soon." An mAP of 0.35 is still far from the 1.0 upper bound; "far beyond baselines" is not "good enough." The open-source repo RoboTok-Code (MIT license) does not include the 100K-video corpus: dataset-building code depends on a private clip database that is not shipped, and the MANO/SMPL-H hand models require separate registration — reproducing the "internet-scale" pipeline end to end means assembling your own corpus first.
So what
The embedding-plus-vector-search infrastructure of the LLM era is migrating into the data layer of embodied AI. When humanity's video libraries can serve as robot textbooks, the bottleneck moves from collection to retrieval quality and real-world closure. Whether these numbers survive contact with real robots is the paper's next question.
Refs: arXiv:2609.03199 / rice-robotpi-lab.github.io/RoboTok/