RAG pipelines usually carry a hidden cost: after retrieval, a dedicated reranker encodes every query-passage pair, and then the generator reads the passages again. Two models stay resident, and the candidates are processed twice. A paper posted to arXiv on October 29 by Airun Intelligence Labs, titled Glance, asks a blunt question: since the generator is going to read these passages anyway, why not let it score all of them itself? (paper)

The mechanism: one shared prefill, one short probe per passage

Glance attaches a rank-16 LoRA adapter and a 256-dimensional pointer head to a frozen 9B backbone (MiMo-V2.6-Distill-Qwen-9B, a hybrid that interleaves Gated DeltaNet linear-attention layers with full-attention layers at 3:1). Those are the only trainable parameters. All k candidate passages are packed into one shared context; a roughly 30-token relevance probe branches from the shared cache for each passage, and each probe yields one score. Passages are prefilled once, never encoded pair by pair.

Because every probe sees all candidates, the authors suggest it can compare a passage against its neighbours, which isolated scoring cannot do. Once scoring is done, the adapter is switched off and the same backbone becomes an ordinary chat model again: mode switching took 4.2–9.4 ms in their tests with bit-identical probabilities, resident memory is 16.85 GiB, and each additional decision skill costs about 0.1 GiB.

A leaner variant activates the adapter only on probe tokens: passages are encoded with the base weights, so the answer can continue straight from the reranking cache and passages are truly read once. On 100 HotpotQA questions, the shared state pass took 64 ms and ten probes 85 ms; continuing the cache answered slightly faster than re-prefilling the top-3 (138 vs 181 ms) at essentially the same F1 (0.750 vs 0.754). The authors flag this as a 100-question, single-seed result.

The numbers: 39 minutes of training, self-reported scores slightly above a cross-encoder

After 39 minutes of fine-tuning on MS MARCO (4,000 queries, 55,598 labelled probes, one B200, 21 GiB peak), the paper reports nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, against 0.721 and 0.326 for the dedicated cross-encoder bge-reranker-v2-m3; 14 more minutes on longer records lift the pair to 0.749 and 0.346.

The authors apply their own brakes: BEIR differences are not significance-tested, and with roughly 300 queries, gaps below 0.02 nDCG@10 should be read as indicative. The 0.741-versus-0.721 lead, by the paper's own standard, deserves a cautious reading.

The downstream result carries more weight. On 300 HotpotQA questions, the same 9B generator answering from Glance's top-3 passages scores F1 0.729, versus 0.684 from the cross-encoder's top-3, a gap of +0.045 (paired bootstrap 95% CI [0.014, 0.076]), with higher gold-paragraph recall as well (R@2 0.897 vs 0.863).

The counterintuitive bit: lower recall, better answers

In the zero-shot experiments there is a finding worth remembering: Glance's gold-paragraph recall is clearly below the cross-encoder's (R@2 0.762 vs 0.863), yet the generator answers better from its selection (F1 0.718 vs 0.683). The authors' reading is that the generator picks context it can actually use — followed immediately by another brake tap: the training labels are passage relevance, not answer quality, so this does not show that the model has learned answer utility.

An honest list of negative results

  • Speed: the dedicated cross-encoder runs 31–60 ms per query; trained Glance takes about 420 ms on a B200. The paper and the model card both state plainly that a cross-encoder remains faster per query.
  • Memory: every probe branch currently copies the shared cache, so memory grows with candidates times context length — a 56 GiB peak at 100 candidates; 200 candidates (43.4k tokens) do not fit in one call and take 15.8 s per query on a chunked path.
  • The zero-shot adapter degrades past 20 candidates and collapses to 0.505 at 200; the trained adapter instead keeps improving with k (0.717 at 200 candidates, vs 0.659 for the cross-encoder).
  • Batching across queries currently lowers throughput (2.87 down to 1.57 queries per second).
  • Specialisation forgets: on the general decision benchmark JevBench, the reranking adapter scores 175/231 versus 184/231 for the general adapter, with 7 of the 9 lost items in the hard tier.
  • Passage order matters: nDCG@10 moved between 0.678 and 0.693 across passage orders, with the top-ranked passage unchanged in 91% of cases — the price of probes that see their neighbours.

The limitations section is unusually direct: this is a proof of concept, evaluated on two retrieval datasets, one QA dataset and one backbone family. On the engineering side, a vLLM prototype serves chat and decision rows in one engine, with chat requests running adapter-free and decision rows carrying the adapter over a shared prefix cache: across 20 packed requests its outputs track the research runtime at Spearman 0.995, with throughput of 1.54 versus 0.86 requests per second — a production path exists, but probabilities still deviate by 0.006 on average.

So what

The valuable part is not the 0.741 score but the accounting: a second resident reranker is a real memory and processing cost, and Glance offers a LoRA-scale alternative — one backbone, several 0.1 GiB decision skills, switched per request. The speed and memory engineering has not been done, so cutting the reranker in production today would be premature; adapters and code are open-sourced under Apache-2.0 on Hugging Face. For anyone running RAG services, the more practical lesson may be that counterintuitive finding: the passage relevance a reranker optimises and what the generator actually needs might not be the same objective at all.