On June 28, the OMLab team at Hugging Face open-sourced VLX-Seek, a 3B-parameter visual-language model focused on fine-grained visual understanding. Its core innovation: replace the "predict a bounding-box coordinate" paradigm with "region reference" — the model directly outputs image-region references (patches, polygons, scribbles, masks), removing the heavy decoding layer traditional detection heads add to VLMs.
The technical details: VLX-Seek uses a unified region tokenizer that compresses image regions of any shape into a fixed-length token sequence, then works with the standard VLM next-token-prediction training objective. The model can simultaneously output language descriptions and region references, and supports flexible region shapes (rectangles, polygons, scribbles, masks), breaking free from the "must predict four numbers" constraint of traditional detection.
The highlight is performance: on 11 fine-grained perception benchmarks, VLX-Seek-3B is on par with or surpasses Gemini 3.1 Pro on most tasks, with particularly notable results on RefCOCO/+/g, Visual Genome relation understanding, and OCR-with-region tasks. The 3B model can be deployed on a single consumer GPU, while the closed-source Gemini 3.1 Pro requires cloud APIs.
VLX-Seek's value is not just the SOTA — it proves that fine-grained visual understanding doesn't have to rely on "predict four numbers" or "generate full masks." Region reference, as a more flexible intermediate representation, is more suitable for VLM's autoregressive paradigm, and may become a new direction for multimodal models' visual grounding.