OpenAI and Broadcom unveiled Jalapeño, an LLM-inference-optimized custom chip. The standout: 9 months from initial design to tape-out — a record for a custom AI accelerator — and the chip is already in volume production at Broadcom's fabs.
The technical details: Jalapeño is a 7nm ASIC with 144GB HBM3e and a custom tensor-core design optimized for LLM decode workloads. The "inference-only" focus is key — Jalapeño sacrifices training flexibility for inference efficiency. The architecture includes a "speculative-decoding accelerator" (a hardware block for verifying draft tokens), a "KV cache compression engine" (hardware for 4-bit/2-bit KV cache), and a "low-latency interconnect" (for multi-chip inference of large models).
The performance: on Llama-3-70B inference, Jalapeño hits 2.3× the tokens-per-watt of NVIDIA H100. On a 256-chip cluster, Jalapeño can serve a 1T-parameter model with sub-100ms per-token latency. The chip is also "NVIDIA-compatible" — it runs the same CUDA software stack, with minimal code changes.
The strategic angle: Jalapeño is OpenAI's answer to the "inference is the new bottleneck" problem. As OpenAI's API traffic grows, the inference cost becomes the dominant expense, and a custom chip can cut it by 50%+. The Broadcom partnership gives OpenAI access to Broadcom's networking and packaging IP, and Broadcom gets a flagship AI customer to anchor its custom-AI business.
The bigger takeaway: the "custom AI inference chip" race is heating up. Google has TPU, Amazon has Trainium, Meta has MTIA, Microsoft has Maia — and now OpenAI has Jalapeño. The era of "all AI workloads run on NVIDIA" is ending, and each hyperscaler is building its own custom silicon.