LongCat-Next Turns Text, Images, and Audio into the Same Next-Token Prediction Problem
Most multimodal systems are unified only at the interface. Images pass through a vision encoder, audio goes through a dedicated audio module, and their features are eventually connected to a language model. LongCat-Next, open-sourced by Meituan’s LongCat team, takes a more radical and structurally simpler route: it discretizes text, vision, and audio into a shared token space, then uses one autoregressive objective for understanding, generation, and conversation.
The project calls this method DiNA, or Discrete Native Autoregression. Instead of building a separate backbone for every modality, it equips each modality with tokenizer-detokenizer pairs and extends the established training infrastructure of large language models to visual and audio signals. The backbone is LongCat-Flash-Lite MoE, described by the project as an A3B model, and the system is positioned as a native multimodal foundation model that can see, create, and talk. The project description and code are available on GitHub.
Why discrete vision is the key problem
The difficult part is not merely converting an image into tokens. The real challenge is preserving both what an image means and what it looks like inside those discrete representations. LongCat-Next combines Semantic-and-Aligned Encoders (SAE) with Residual Vector Quantization (RVQ) to build hierarchical visual tokens. SAE supplies semantic abstraction, while RVQ retains fine-grained visual information.
The team also introduces dNaViT, the Discrete Native-Resolution Vision Transformer. It treats visual features as discrete “visual words,” supports dynamic tokenization and detokenization, and works with native image resolutions. According to the project, visual understanding and visual generation do not need to remain two architecturally separate systems. Both can be reformulated as different outputs of the same predictive process.
The engineering implication is direct. The model is no longer just a language model with a vision plugin attached. Instead, multiple modalities enter the same discrete embedding space. The README states that the model maintains strong generation quality even at a 28× visual compression ratio, with particular emphasis on text rendering. On the audio side, it covers speech understanding, low-latency voice conversation, and customizable voice cloning.
Open source means more than releasing weights
LongCat-Next releases both model weights and source code under the MIT License. The repository includes examples for text, image understanding, image generation, audio-to-text, audio generation, and speech synthesis. It also provides supervised fine-tuning code built on PyTorch FSDP2. The SFT pipeline supports image-plus-text to text, text to image, and unified mixed-sample training.
The deployment requirements are explicit. The Transformers setup needs at least three GPUs with 80GB of VRAM each, with H100 and A100 80GB given as examples. The recommended environment includes Python 3.10 or newer, PyTorch 2.6 or newer, and Transformers 4.57.6 or newer. The project also provides basic SGLang adaptation, while more complete deployment support lives in a separate inference repository.
That makes LongCat-Next more of a research and industrial validation foundation than a consumer model that runs with one click. Its importance is not limited to any single benchmark. The deeper question it tests is whether discrete tokens can become a common language for text, images, and audio.
If that direction continues to work, multimodal system design may shift away from continuously adding specialized modules. The competition would instead move toward better tokenizers, stronger shared representation spaces, and more effective unified training. LongCat-Next gives a clear answer: the next step in multimodality may not be connecting more components, but turning every modality into “words” the model can genuinely internalize.