AI capability is downstream of hardware in a way that is unusual for software. The limiting factor on what gets built is not ideas but access to accelerators, to high-bandwidth memory, to advanced packaging capacity, and — increasingly the binding constraint — to electricity. Most of the industry’s stranger strategic behaviour becomes legible once you see the stack.
Why GPUs, and why they are not really GPUs
Training and running a neural network is, almost entirely, large matrix multiplication. That workload is enormously parallel — thousands of independent multiply-accumulate operations — which is exactly what graphics hardware was built for. The architectural match was a historical accident that became a trillion-dollar industry.
Modern data-centre accelerators have drifted far from graphics. They carry dedicated matrix units, operate at very low numerical precision (8-bit and 4-bit formats are now standard for inference, where a decade ago 32-bit was assumed), and in at least one current generation have had their double-precision floating-point capability cut dramatically — an explicit statement that these are not general scientific computing parts.
The other shift is that the unit of sale is no longer a chip. It is a rack: dozens of accelerators plus host CPUs, wired together with proprietary high-speed interconnect and liquid cooling, sold as one system costing several million dollars. Because frontier models do not fit on a single accelerator, the interconnect between them is as much a competitive moat as the silicon.
Memory, not compute, is usually the bottleneck
This is the most important and least intuitive fact in the stack. For inference especially, accelerators commonly sit idle waiting for data rather than running short of arithmetic. Generating each token requires reading model weights from memory, so throughput is governed by memory bandwidth.
Hence high-bandwidth memory — DRAM stacked vertically and placed beside the processor — and hence HBM’s position as the real chokepoint. Current generations are effectively sold out ahead of production, supply is concentrated among very few vendors, and one supplier holds the majority share. A great deal of architectural effort goes into reducing memory traffic rather than adding compute: quantisation, attention variants that shrink the cache, and sparse models that activate only a fraction of their parameters per token.
The second chokepoint is advanced packaging — the technology that places logic and memory stacks on one substrate. Capacity is booked out, lead times run well over a year, and a single buyer holds a majority of available allocation. You can design an excellent accelerator and be unable to have it assembled.
The competitive landscape
One vendor holds a dominant share, and the durable part of that advantage is software: well over a decade of accumulated libraries, kernels and developer familiarity that competitors must match before their hardware specifications matter at all.
The credible challenges come from three directions. The main GPU competitor has landed multi-gigawatt commitments from several frontier labs, which is the relevant signal — not specifications but whether serious buyers will commit. The hyperscalers have all built custom silicon, generally inference-focused and generally for internal use, which is best understood as negotiating leverage and margin recovery rather than an attempt to sell chips. And a cohort of specialist startups targets inference specifically, where a different architecture — very large on-chip memory instead of external stacks — can beat general-purpose accelerators on latency. That category has seen both a major acquisition and a substantial public listing recently, with the clear majority of new funding in the sector flowing to inference rather than training.
The economics flipped to inference
Training costs are estimates — no lab discloses them — but credible analysis puts frontier programmes in the high hundreds of millions to low billions, with compute the dominant line item, and finds training cost growing at roughly two and a half times per year.
The more consequential shift is that inference now consumes the majority of AI compute, having been roughly a third a few years ago. The reason is simple arithmetic: training happens once, inference happens on every request forever. For a widely-used model, cumulative serving cost overtakes its entire training cost within months.
This explains a lot: why reasoning models, which deliberately spend more compute per query, are economically significant rather than merely interesting (see reasoning models explained); why every vendor offers steep discounts for batch processing and cached prompts; and why inference-specialised hardware attracts disproportionate investment.
Power is now the limit
Industry analysts have moved from describing AI as chip-constrained to describing it as power-constrained. Data-centre electricity demand is growing at well over twenty per cent annually, with AI-optimised servers the fastest-growing component and on track to overtake conventional servers. Projections put data centres at a low single-digit percentage of global electricity within a few years.
At the individual site level the numbers have become genuinely unusual: the largest operational AI campuses draw on the order of hundreds of megawatts to approaching a gigawatt, with announced end-states beyond two gigawatts. For scale, that is a mid-sized city’s consumption for one building complex.
The consequences are physical and political. Grid interconnection queues are long enough that several major projects have added on-site gas generation to bypass them. Local opposition and permitting delays have blocked or stalled projects worth well over a hundred billion dollars. Announced capacity and delivered capacity have diverged substantially — headline programme figures should be read as intentions, with only a fraction typically live.
Export controls and the second ecosystem
Because compute is the input to capability, chips became foreign policy. US controls restrict advanced accelerator exports to China, with the permitted tier shifting periodically and top-end parts consistently excluded.
Two developments complicate the simple story. China has restricted imports of the very chips the US permits, steering domestic firms toward local silicon — so partial relaxations have had limited commercial effect. And domestic Chinese accelerators now land credibly between recent Western generations in capability, including custom memory technology designed specifically to route around HBM controls. The binding constraint on that ecosystem is manufacturing capacity and yield rather than design.
The durable consequence is a genuine second hardware and software ecosystem, which was not the intended outcome.
What to take from this
Three things. Hardware supply sets the ceiling on AI progress more directly than research does, which is why compute commitments are the most informative signal in the sector. Memory and packaging, not arithmetic, are the real scarce resources. And power has become the constraint that cannot be solved with capital alone, because it requires grid infrastructure on timelines that do not compress.
Last reviewed: September 2026. Hardware generations, pricing and capacity figures move quickly, and vendors do not publish rack pricing — cost figures in this area are analyst estimates with wide spread.
