At Hot Chips 2026, OpenAI released the first batch of measured data for its in-house inference chip, Jalapeño. The chip is a general-purpose inference accelerator co-developed with Broadcom, aimed squarely at the part of AI compute that is truly tight today: the user-facing response loop.
Hot Chips first results: 1.5–1.9× per watt, latency compressed to ~1 second
The benchmark is SemiAnalysis's InferenceX. Jalapeño turned in a stable lead across three public models: on GPT-OSS 120B (nominal 8k/1k), peak mixed TPS per kilowatt ran roughly 1.9× the GB200 comparison, with end-to-end latency falling from 1.80s to 1.03s; on DeepSeek R1 670B (MXFP4), peak per-watt was 1.7× and latency dropped from 5.99s to 1.65s; on Kimi K2.5 1T, peak per-watt was 1.5× and latency dropped from 5.31s to 1.56s. Across the full operating range, from ultra-low-latency conversation to high-throughput batch, Jalapeño sits on the Pareto frontier.
Beyond the numbers, the architectural choices matter more. OpenAI positioned the chip as "built for LLM inference from day one," with different resource balances for the three bottleneck phases of inference: prefill is compute-heavy, decode is constrained by memory bandwidth, and inter-chip communication can leave compute units idle while waiting on data. Jalapeño tackles all three with a single architecture: KV cache is placed explicitly and kept local, and a large-domain network keeps the entire request inside one connected system, reducing data movement and synchronization overhead.
AI-written kernels, nine months from design to tapeout
Another detail worth flagging is AI's place in the chip development loop itself. OpenAI used its own models to compress the design, verification, and iteration cycle — from project kickoff to tapeout took only nine months. The team also made the hardware a programming target that "both humans and AI can use": with Codex plus GPT-Astra for kernel generation, AI-produced kernels for selected GPT-OSS attention and MoE blocks ran 1.5–1.8× faster than human-expert-written versions. The figures apply to selected blocks, not the full model, but the direction is strong: AI is not only being served by inference chips, it is becoming the compiler of chip development in return.
Timeline and what it actually means for Nvidia
Hardware lead Richard Ho set the cadence: Jalapeño deploys at the end of 2026 in "very small volumes," with meaningful scale coming in 2027; Gen 2 is deep in development and Gen 3 is already taking shape. That timing lines up with Nvidia's Hot Chips talk on the same week framing power supply as the binding constraint — both companies have made watts the headline knob. Notably, OpenAI also reiterated it "is not decoupling from Nvidia": it will keep deploying Nvidia and other partner accelerators broadly for both training and inference. Jalapeño is not a replacement, it is a complement — filling exactly the lowest-cost-per-watt slot in Nvidia's stack, the slot that gets hammered once a model is in production and being called over and over.
So what: once silicon is a solved game, the real race is "per watt"
The signal for practitioners is clear: the next stage of AI compute is not about "who has the bigger GPU," it is about "who can squeeze more tokens out of every watt." Agentic workloads, long context, and real-time audio/video — the latency- and sustained-power-sensitive regimes — will be the first to feel the benefit. Conversely, players leaning on "more cards for more intelligence" are seeing their marginal returns visibly compress.
References: Jalapeño first results — OpenAI, OpenAI's Jalapeño chip — TechCrunch