Scrolling model leaderboards leaves one question unanswered: can the model actually interleave text and images into a coherent piece. A paper posted to arXiv at the end of October (arXiv:2610.38425) breaks that into seven measurable dimensions, and the result is counterintuitive. The closed-source flagship wins big on pure text, yet on the two dimensions that define interleaving, it trails the open-source models.
Why interleaved generation has been hard to evaluate
Interleaved generation means a model emits a continuous sequence where text and images alternate, like a recipe with a photo next to every step. This is not one image plus one caption: the model has to hold narrative progression, image count, and whether each image matches the preceding context all at once.
The paper states the situation plainly. Unified Multimodal Models (UMMs) can already jointly generate text and images, but interleaved generation remains "largely underexplored." Two blockers are named: the lack of well-defined interleaved generation tasks with corresponding training data, and the absence of fine-grained evaluation protocols that can reliably diagnose model performance across capability dimensions.
The gap on the data side is starker. The authors collected a million-scale raw corpus of interleaved text-image content from seven sources that permit public collection and redistribution. Only about 10 percent survived: if any segment of a long interleaved sample is defective, the whole sample goes.
What IntReal ships: 100K samples plus a 600-sample suite
The surviving IntReal-100K contains 100,000 samples, 538,452 images, and 146.6M text tokens, averaging 5.38 images and 1466.06 text tokens per sample. The data splits into four categories by how text and images collaborate: Planning (decisions), Knowledge (explanation), Tutorial (ordered steps), and Story (narrative).
The seven sources contribute 34,771 samples from Xiachufang, 21,138 from Wikipedia, 21,055 from WikiVoyage, 5,055 from Wikitravel, 7,298 from Curious George, 3,949 from Rabbids Invasion, and 6,734 from The Land Before Time. Length distribution tells its own story: Story averages about 8.6 images per sample, the longest interleaved sequences, while Planning averages only 2.42 to 2.68, with text carrying decisions and images visualizing the spatial and temporal changes.
The evaluation set draws 600 samples by stratified sampling, 150 per category, in two granularities. Holistic evaluation has the model generate the complete interleaved response at once; fine-grained evaluation pins the preceding text-image context and asks only for the next image, so you can localize which step went wrong. The cost is steep: each model has to generate more than 6,000 images interleaved with text.
Each of the seven dimensions gets a dedicated expert model. Text Richness and Text Logicality go to an LLM judge running Gemma4, Image Aesthetic to the RALI IQA model, Image Diversity to DINOv3 similarity between adjacent generated images, Quantity Compliance to plain image counting, Context Fidelity to a VLM judge, and Subject Consistency to DINOv3 comparison against reference images.
Can the evaluation itself be trusted
The authors ran three checks. Metrics do not overlap: the absolute pairwise sample-level Spearman correlation between any two dimensions stays below 0.42, with the most strongly correlated pair being Context Fidelity against Subject Consistency. It agrees with people: across five subjective dimensions they collected 750 expert pairwise comparisons, with automatic scores matching expert judgments 83.3 percent of the time on average. It survives a different judge: swapping Gemma4 for the similarly sized open-source Qwen3.6-35B-A3B keeps model rankings correlated above 0.9 (Text Richness 0.905, Text Logicality 0.929, Context Fidelity 0.976).
The results: closed source wins on text, loses on interleaving
Four UMM families were brought in: Bagel, Emu3.5, SenseNova-U1, and the closed-source API NanoBanana2, which the paper labels Gemini3.1-Flash-Image.
The outcome is not one model dominating, it is capability profiles that do not line up. NanoBanana2 posts 90.80 on Text Richness and 97.49 on Text Logicality, while the leading open-source Text Logicality score, Emu3.5, reaches 75.17. But on the two dimensions tied to interleaving, it falls behind: Image Diversity 83.56 against an open-source high of 93.31, and Context Fidelity 76.70 against an open-source high of 87.29. The relative deltas in the table put Text Richness 222.33 percent above the open-source high, Image Diversity 10.45 percent below, and Context Fidelity 12.13 percent below.
One detail in reading that table is worth flagging. The open-source comparison is a per-metric envelope across checkpoints, not one single model, and that envelope already includes checkpoints specifically fine-tuned on this very dataset. The closed-source API is therefore measured against open-source models that had already studied for this test.
The trade-off post-training exposes
The authors also checked whether their own metrics can work as training signal. Fine-tuning SenseNova-U1-8B-MoT on IntReal-100K lifts Text Richness by 29.04 percent, Text Logicality by 6.68 percent, and Subject Consistency by 6.53 percent, but costs 5.65 percent on Image Aesthetic and 4.89 percent on Context Fidelity, leaving the seven-dimension mean up just 0.60 percent. That lines up with the widely reported optimization conflict between understanding and generation inside UMMs.
Reinforcement learning on top (Flow-GRPO with IQA, DINO, and rubric rewards over 10K carefully annotated samples) pulls Image Aesthetic back from -5.65 percent to -1.34 percent, turns Image Diversity positive at +5.60 percent, and lifts the mean to +2.69 percent, the top score among open-source models.
The broader conclusion from the authors matters more than the deltas: improving unimodal generation capability generalizes to interleaved generation, so interleaved generation does not require joint text-image training at every stage, and modality-specific optimization is a viable path. Training and evaluation are strictly disjoint: 99.4K training samples, with the 600 evaluation samples removed entirely.
A few cold-water caveats
The stated limitations in the paper: high-quality real interleaved data requires aggressive filtering, which keeps the retention rate low, and to keep the experimental setting clean the authors did not explore mixing in unimodal data for post-training, leaving synthetic data augmentation and mixed-data post-training for future work.
Several things are worth weighing independently. Human agreement averages 83.3 percent, meaning roughly one in six pairwise comparisons has the automatic score disagreeing with the expert judgment, so this benchmark ultimately has models grading models. The closed-source side is more delicate: the paper explicitly notes that because the system is not publicly disclosed, the authors cannot determine whether the interleaved capability of NanoBanana2 is natively supported or implemented through an agentic pipeline, so reading "closed source loses to open source" needs a discount. Licensing is handled source by source: all Story material comes from Curious George, Rabbids Invasion, and The Land Before Time, with each image licensed individually, and user-contributed Xiachufang content requires attribution and is limited to non-commercial use.
One engineering reality: the whole evaluation runs on a single 80GB GPU such as an A100 80GB or H100 80GB. That is not a high bar, but it explains why the evaluation set stops at 600 samples, since the authors note that scaling further would substantially increase evaluation cost across multiple model families.
So what
For anyone building models, the value here is not another leaderboard. It is that "interleaving ability" gets decomposed from a vague impression into seven mutually distinct, reproducible dimensions that can even serve as training signal. Being strong at text is not the same as being strong at interleaving, and a lead in the unimodal modality gets erased in text-image alternating scenarios. That is the most counterintuitive finding in these results.
On October 30, the authors put the dataset, the leaderboard, and the fine-tuned models on Hugging Face simultaneously, with the code repository open alongside. Anyone reproducing this can start running; anyone wanting the training data is welcome to it, since it was prepared for training in the first place.