OpenAI's Jalapeño chip is built for fast inference at scale, benchmarks show

First, understand Jalapeño's positioning. It is not a training chip for building large models — it is designed specifically for inference, the computing that actually runs the model and generates answers in the background when a user asks ChatGPT a question. As ChatGPT's user base has grown exponentially, inference cost has become one of OpenAI's largest expense items. Ho stressed in the press call that Jalapeño has two design goals: serving more AI workloads per unit of power, and returning responses with very low latency. In other words, OpenAI wants this chip to support high-concurrency access from massive numbers of users while keeping each user's wait time short. In the InferenceX benchmark, Jalapeño outperformed Blackwell on both dimensions. Ho acknowledged, however, that the comparison has a time gap — Nvidia's Blackwell is currently shipping hardware, while Jalapeño will not deploy until late 2026 'in very small volumes', with significant deployment coming in 2027. By then, Nvidia's next-generation products may also be in place, and the competitive landscape may not look the same as today.

The R&D backdrop of Jalapeño is equally noteworthy. The chip was first announced in October 2025, developed by OpenAI in close collaboration with Broadcom, and OpenAI's own models assisted in the chip's development — using AI to design AI hardware is itself an expression of full-stack innovation. OpenAI plans to make Jalapeño a multigenerational platform, allowing AI products, models, chips, and memory to be developed in concert. Because of this full-stack approach, OpenAI was able to target the phases of inference that often create bottlenecks: Jalapeño is specifically designed to minimize delays during the prefill phase — the most time-consuming part when a model processes a prompt — and the communication phase, the overhead of data exchange when multiple chips work together. Ho said OpenAI hopes this software-hardware co-design will let Jalapeño serve customers at scale efficiently while keeping latency low, significantly reducing unit compute cost without sacrificing experience.

From an industry-structure perspective, Jalapeño's debut is a landmark in the wave of AI compute autonomy. Over the past two years, nearly every leading AI company has been building custom silicon to escape dependence on a single chip supplier and cut soaring compute bills: Google has TPU, Amazon has Trainium and Inferentia, Meta is advancing its own chip project, and Microsoft is developing custom hardware with OpenAI. As the company most heavily dependent on Nvidia, OpenAI's chip progress has long been seen as a bellwether. Publishing these benchmarks sends several key signals: first, custom inference chips can now compete head-on with the traditional champion on efficiency; second, software-hardware co-design is becoming the new high ground of AI infrastructure competition; third, energy-efficiency metrics like throughput per kilowatt are replacing raw compute numbers as the new yardstick for AI chips — because in an era of soaring electricity costs, efficiency is cost, and cost is competitiveness. Of course, there is a long road from benchmarks to large-scale stable deployment, and whether Jalapeño can truly carry ChatGPT's inference load in 2027 remains to be seen. But either way, the era of 'Nvidia dominance' is being rewritten, chip by chip, by these challengers.