For three years the default answer in LLM pretraining has been "more tokens, more compute." A technical report submitted to arXiv on Sep 9 proposes a different route: while predicting the next token, the model also learns to predict the "next concept." NCP-ArchPreview climbed to #1 on Hugging Face's daily paper ranking on Sep 11 with 127 upvotes — the community is clearly intrigued.
Beyond tokens: predicting concepts
The core change adds a parallel concept-level objective, Next Concept Prediction (NCP), on top of standard next-token prediction (NTP). The model builds a product-quantized concept vocabulary directly from its own hidden states, encoding semantic units that span multiple tokens as discrete concepts; a dedicated Concept Module predicts future concepts, and those predictions are fed back to the token level to guide subsequent generation. The two objectives are trained jointly end-to-end, while generation remains standard token-level autoregression — no separate inference paradigm is introduced.
Half the tokens, the same loss
The scale numbers are the hardest part of this report: the architecture is scaled to 8.9B parameters and trained on 5.73T tokens from the Dolma-3 dataset — per the authors' own claim, the largest demonstration of a latent-space language model to date. On results: consuming only 51.3% of the total training tokens, NCP-ArchPreview reaches the final pretraining loss of OLMo-3-7B; after full pretraining it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a 5.99-point gain on GSM8K. Controlled experiments isolate the gains attributable to the latent architecture and to the NCP objective separately; against a strictly parameter-aligned 8.9B baseline, it approaches that baseline's training loss using 85% of the computation.
The latent space keeps working after pretraining
The concept space stays useful once pretraining ends: updating just the 17M-parameter VQ module yields a lightweight interface for domain adaptation, and injecting concept representations into a DFlash2 speculative-decoding drafter improves mean accepted length by 4.17% with negligible overhead. In other words, the concept layer doesn't just serve the training objective — it feeds back into inference acceleration and domain transfer.
"Preview" is in the name for a reason
A sober read: the model is called ArchPreview for a reason. The comparison baseline is the open OLMo-3-7B, not frontier closed models; "largest latent-space language model" is the authors' own claim, and independent replication will take time. The paper is signed by the 28-author Intern-NCP Team (including Dahua Lin, Zhouhan Lin, and Bowen Zhou), and the Hugging Face page already lists 18 models citing the paper — community follow-up is moving fast.
The more expensive compute gets, the more objective-function innovation is worth. As the marginal returns of "double the tokens again" keep falling, making every token carry a harder prediction task may be a cheaper scaling direction than stacking more GPUs. Full paper: arXiv:2609.10715; community discussion: Hugging Face paper page.