A neural architecture search method that reaches 97.24% test accuracy on CIFAR-10 within its original search space collapses to 39.77% when evaluated on NAS-Bench-201, a benchmark built on the same dataset. That is the opening story of a new paper, "Large Language Models as a Universal Searcher for Open-World Automated Machine Learning": today's AutoML tools break when the search space shifts, and they optimize architectures, hyperparameters, and data ratios in isolation. The proposed fix is to train an LLM as a cross-domain universal searcher.

Turn every design choice into numbers

The paper introduces NCode, a unified numerical encoding that maps network structures (which operation sits on each DAG edge), training hyperparameters, and data ratios into discrete numeric sequences. Heterogeneous AutoML decisions thus land in one shared discrete space, and the LLM always sees a string of numbers plus a history of scores. Training data comes from NAS-Bench-301, with roughly 10^21 architecture-performance pairs; stochastic edge pruning at probability 0.5 carves out diverse sub-spaces, yielding about 228,000 optimization trajectories — all derived from a single vision classification task.

Ranking beats generating

The pivotal move is reframing the objective: instead of asking the LLM to generate a full configuration, it receives the historical trajectory plus a candidate pool and picks the most promising one. The ablation is brutal — on ImageNet16-120, disabling the ranking reformulation and forcing direct NCode generation drops test accuracy from 46.51% to 42.53%. The main searcher is a fully fine-tuned LLaMA-3.1-8B with a 20,480-token context window to hold the long history. What the model learned in one domain is simply "how to choose" — no gradient updates on any target task.

Trained on one domain, zero-shot across three modalities

Deployment is entirely zero-shot. On image segmentation, it searches SAM's LoRA design over an architecture space of about 10^19; searched purely in medical imaging, the configuration transfers zero-shot to camouflaged objects, shadows, leaves, and remote-sensing roads, beating the strongest baseline Conv-LoRA by an absolute 2.5 S-measure points on CVC-612. On audio, it handles speaker verification with 3D-Speaker, reaching an average EER of 9.21% versus 9.37% for ERes2NetV2 and 10.20% for OptFormer. The largest campaign is text-to-image: jointly searching the architecture, hyperparameters, and data ratios of a billion-parameter Diffusion Transformer (the paper's conclusion says 1.3B), at a total cost of 896 GPU-days, with the paper reporting a GenEval overall score of 0.91. The ablation shows why joint search matters: architecture-only reaches 0.86, hyperparameter-and-data-only 0.89, and the joint search 0.91.

GRPO-trained CoT opens up the searcher's mind

The team also trains a self-explaining chain-of-thought in Qwen3-8B via GRPO, distilling the searcher's reasoning into six behavioral patterns, and finds a clear strategic shift: early search leans on neighborhood mutation (small perturbations around high-scoring configurations) for broad exploration, then pivots to anchor exploitation. Strategy also adapts to task complexity: on the simpler NAS-Bench-201, the global-component-distribution strategy rises from 10% to nearly 20% usage, while on the heterogeneous text-to-image joint search it stays around 5% — the model "knows" that joint optimization cannot rely on simple counting.

Paper: https://arxiv.org/abs/2611.12791

So what

The durable lesson is not any single leaderboard number but two judgments. First, an LLM's ranking ability transfers across domains far more easily than its generating ability — picking a good answer from a pool is a lower bar than writing one from scratch. Second, the next step for AutoML may not be ever-more-elegant expert algorithms but a searcher pretrained on generic optimization trajectories that works anywhere. Of course, 896 GPU-days shows that "universal" is not cheap, and the method has yet to touch the pretraining search space of LLMs themselves. Once models learn to tune models, how wide is the moat left for the people who tune them?