arXiv 2606.20295 introduces a 62-page survey that decomposes LLM inference into a "four-layer architecture," providing an industry-oriented view of "Token Operations" (the discipline of serving LLM inference at scale). The survey is co-authored by leading inference infrastructure researchers from OpenAI, NVIDIA, Anthropic, and academic institutions.
The four layers: (1) the Model Layer — the neural network itself (Transformer, SSM, etc.); (2) the Scheduling Layer — request batching, routing, prioritization; (3) the Execution Layer — kernel optimization, KV cache management, speculative decoding; (4) the Hardware Layer — GPU, TPU, memory, interconnect. The survey dedicates a chapter to each layer, with a comprehensive analysis of techniques and trade-offs.
The "Token Operations" framing: the survey coins the term "Token Operations" to describe the discipline of serving LLMs at scale — analogous to "DevOps" or "MLOps" but focused on inference. The survey argues that "Token Operations" deserves to be a separate discipline, with its own best practices, tools, and research community.
The "industry-oriented" highlight: unlike academic surveys that focus on algorithms, this survey is explicitly industry-oriented. It covers topics like "how to price LLM inference," "how to manage inference capacity," "how to handle multi-region deployment" — topics that are essential for production LLM services but rarely covered in academic literature.
The bigger takeaway: "Token Operations" is becoming a real engineering discipline. As LLM inference becomes a $100B+ industry, the operational complexity (scaling, cost optimization, multi-region deployment) is exploding, and the "Token Operations" framework is the right way to organize this discipline. For the industry, this means "Token Operations Engineer" will become a new job title, similar to how "DevOps Engineer" emerged in the 2010s.