Autoregressive image models carry one hard limitation: they can only paint from the top-left corner to the bottom-right, one token at a time. Because the token ordering is nailed down to a raster scan at training time, the model cannot fill in a patch in the middle, extend the canvas outward, or repaint a single region at inference time. This is not a case of the model failing to learn the capability — the capability was never written into the weights.
The standard fix is to retrain from scratch, teaching the model a mixture of randomly permuted token orders. The paper cites RandAR, a random-order autoregressive model that needs 300 epochs where the corresponding raster-scan LlamaGen-XL needs 100 — roughly 3x the compute. RandAR's own project page confirms it was trained from randomly permuted token sequences. The other mainstream route, discrete diffusion, also costs about 3x.
A NeurIPS 2026 paper (arXiv:2610.39293) argues that 3x is unnecessary: add two lightweight components to a pretrained model and a finished raster model becomes an any-order generator.
What actually gets saved is deployment compatibility, not compute
The paper's comparison table is explicit: RandAR (from scratch) 3x, discrete diffusion 3x, RAR 4x, while this adaptation recipe costs 1.25x on LlamaGen-XL and 1.13–1.33x across four base models.
But the consequential column is not the training-cost one — it is the three compatibility columns sitting right next to it. RandAR trades 3x for "cannot serve with vLLM". Discrete diffusion costs even more and loses both the KV cache and vLLM: its bidirectional attention rescans every unmasked position at each step, which is structurally incompatible with the per-token cache reuse that AR inference frameworks depend on. The adaptation recipe is the only row with all three checked — any-order, KV cache, and vLLM serving — at the price of a 1.25x training budget. The paper notes that for practitioners who already hold pretrained weights, the marginal adaptation cost is over an order of magnitude lower (12x) than training any-order from scratch.
The technical core is two components, neither of which touches the base model's causal single-stream transformer.
One component is future-aware positional embeddings. Under raster scan, the target position at step t is implicitly known — it is t+1. Once the ordering varies, the same observed context can correspond to entirely different prediction targets: given the rest of one image, predicting a sky token and predicting a face token are two different conditional distributions, so the model receives conflicting gradients simultaneously. The fix is to explicitly tell the model which position to fill next, using target-position-indexed 2D RoPE routing plus an additive embedding table. For LlamaGen-XL that table holds 0.74M parameters, or 0.095%.
The other is an ordering curriculum: rather than switching to random orders in one step, the schedule ramps gradually, laying down a stable optimization path from the pretrained solution to a good any-order basin.
Three negative results worth more than the headline numbers
The paper reports FID 6.21 on inpainting against MaskGIT's 10.44, FID 5.84 on outpainting against 12.17, and LPIPS 0.125 on region reconstruction versus MAR-L's 0.174. It is also notably careful with itself: with the Halton schedule, MaskGIT's inpainting FID drops to 7.82, and the paper concedes that part of the original margin came from the competitor's scheduler choice. On outpainting, MAR-L actually has the higher Precision and Recall; with no run-level uncertainty available, the paper calls the two competitive rather than claiming the lead.
The three counterexamples are what are worth carrying away.
Remove the future-aware positional embeddings and training loss rises instead of falling. As the curriculum ramps up the proportion of random orders, the loss curve climbs. That is exactly what the identifiability argument predicts — the gradient conflict is real.
Remove the ordering curriculum and the model sits half-adapted. Starting immediately at full random order (α=1) yields FID 5.86 on LlamaGen-L, against 4.24 for the complete recipe. The paper's diagnosis is sharp: Recall barely moves (0.51 versus 0.52), so this is a fidelity collapse rather than a coverage collapse — the pretrained representation gets eroded before order-flexible routing can form.
The most counterintuitive one: regularizing toward a frozen raster teacher makes things worse. Adding a KL term (λ=1.0) toward the teacher on raster-ordered examples pushes FID from 4.24 up to 9.62. Preserving the old behavior and learning the new conditional distributions are directly in conflict.
One engineering detail: the extra embedding table's learning rate has to be scaled 10x relative to the rest of the network to converge stably, otherwise it cannot keep pace with the curriculum.
The bill still has to be settled
Adaptation is not free. Across four base models, class-conditional FID rises by 1.17–1.93 relative to the corresponding pretrained checkpoint — LlamaGen-L from 3.07 to 4.24, LlamaGen-XL from 2.63 to 3.91, RAR-L from 1.70 to 3.63. The paper volunteers that this cost is "not necessarily fundamental or monotonic with scale", and notes that RAR sits closer to the FID floor, where the same nominal degradation is more visible.
One easily overlooked control: at matched total epochs, adapting a pretrained model clearly beats training from random initialization (LlamaGen-L 4.24 versus 5.60; LlamaGen-XL 3.91 versus 4.81). But the paper reminds us this control matches compute, not schedule — only the pretrained pathway begins with raster-only training.
Inference is a separate line on the bill: any-order training makes parallel decoding possible, cutting forward passes by 9x on LlamaGen-XL, at the cost of FID moving from 3.91 to 5.03.
The boundaries are stated plainly
The work is confined to class-conditional generation on ImageNet at 256×256. Whether the recipe transfers to text-to-image autoregressive models, or to higher resolutions, remains untested. The editing evaluation covers region completion only, and does not measure instruction following or identity preservation. The adapted model retains vLLM serving through a custom model registration, the code is released under the MIT license, and checkpoints are available under the orderar organization on Hugging Face.
So what this advances is not a new model but a narrower and harder claim: different factorizations of the same joint distribution can be bridged by one lightweight adaptation, with no need to train a model from scratch. And the three negative results it leaves behind may be worth remembering more than the tidy FID numbers — especially the teacher-regularization experiment that pushes FID from 4.24 to 9.62.