The first instinct in AI search engineering is to throw GPUs at the problem. A new arXiv paper from the Pollinations team argues otherwise: get caching right, and a single 8-core CPU server is enough to power an open-source AI search engine with session memory and semantic deduplication. The paper, "A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware" (arXiv:2609.05463), centers on OreoLook (formerly lixSearch): local search, caching, session management, and the embedding stack all run on commodity CPU hardware, while answer synthesis is routed to a remote inference provider.

Why Caching Is the Cost Bottleneck of AI Search

OreoLook is an answer engine built on automated browser agents and provider-routed LLM inference. When a question arrives, search agents fan out across the web, YouTube, and images, extract full text and transcripts, and an LLM synthesizes a sourced answer. The authors name three pain points that emerged as usage grew: sessions lost context, equivalent queries triggered redundant work, and URLs were repeatedly embedded across sessions. The common thread: money spent recomputing what was already known.

Three Layers, One Job Each

The core design is a three-layer cache, each layer targeting a specific waste source:

  • Session Context Window: recent conversation turns stay in Redis, with overflow archived to disk using Huffman compression. A background LRU eviction daemon migrates idle sessions from Redis to disk and re-hydrates them on demand, so conversations can resume hours or days later under the configured retention policy.
  • Semantic Query Cache: matching on embedding vectors via cosine similarity catches rephrased queries, skipping the entire search-and-synthesis pipeline. The repo README gives the implementation parameters: Redis DB0, cosine threshold 0.90, repeat queries resolving in under 15 ms.
  • URL Embedding Cache: deduplicates embedding computations for URLs encountered across different sessions.

The deployment numbers come straight from the paper: a single 8-vCPU Intel Cascade Lake server (2 GHz, 32 GB RAM) running 30 Hypercorn worker processes across three containerized replicas, reporting an 89.3% aggregate Redis keyspace hit rate, 0.1 ms read latency, and just 1.38 MB of Redis memory overhead.

Reading Single-Source Numbers Honestly

One point deserves attention: 89.3% is a Redis keyspace hit rate, and the authors explicitly distinguish keyspace hit rate from end-to-end query avoidance in the community discussion—a rare bit of honesty in production-system papers. The repo also packages the caching layer as a standalone PyPI library, lix-open-cache, which depends only on Redis, numpy, and loguru, with no server required; the full search engine ships as a Docker image, self-hostable with a single docker compose command. On the tech stack side, retrieval runs on Qdrant, browser automation on Playwright, the API layer is OpenAI-compatible, and a stateless MCP endpoint supports deep research and PDF exports.

So What

For teams building AI search products, the reference value of this paper lies not in models but in the ledger: big players absorb synthesis compute costs in inference clusters users never see, while under OreoLook's paper-narrow metric, nearly nine in ten keyspace reads are served by cache at 0.1 ms each. The semantic query cache is the layer most worth copying—a rephrased question hitting cache is almost free latency optimization in any multi-turn product. Paper: arxiv.org/abs/2609.05463; code at GitHub pollinations/search.elixpo.