Most Companies Say Their GPUs Sit Idle Half The Time, And They Are Buying More Anyway

Posted on by
Contents show
A flat vector illustration of five identical grey server towers with one glowing warm amber and the others dark, and a small teal coin floating above the glowing one, representing idle GPU capacity and GPU utilization

Nearly four in five companies running AI in production say the GPUs they use sit idle at least half the time, and seven in ten of those same companies plan to adopt a new, additional or replacement AI compute platform within twelve months anyway. Both numbers come from the same pooled survey programme, and both have survived three consecutive months without moving, which is what makes the pairing worth an article rather than a headline: GPU utilization this low is not a passing anomaly in the AI buildout, it is a stable operating pattern with a coherent explanation. That pairing, wasted capacity alongside continued expansion, is the most useful data point published this autumn about whether the AI buildout makes sense, because it moves the question away from the datacenter giants and into the ordinary enterprise budget.

The figures come from three monthly reader surveys pooled by a technology news outlet's research arm, covering 385 qualified respondents at organisations with 100 or more employees, of whom 218 reported GPU utilization. Across the July, August and September waves, the share reporting GPUs busy half the time or less held between 77 and 81 percent each month. Because that research operation is itself the subject of the supplied story and cannot be cited here under this article's sourcing rules, the tables below are presented with their denominators intact so every figure can be weighed on its own sample, and the methodology caveats are carried in full in the limitations section. This article uses those survey tables as its spine, adds the independent measurement from Cast AI's Kubernetes optimisation report, and then does what the survey data alone cannot: explain the mismatch and list what to do about it before the next platform contract is signed.

The numbers, with their sample attached

The headline table deserves reading in full, because each row has a denominator that changes its meaning.

GroupShareCount
Companies running AI in production whose GPUs are busy half the time or less79%172 of 218
Companies with idle GPUs that plan a new platform within 12 months70%120 of 172
Companies with busy GPUs that plan a new platform within 12 months74%34 of 46
Idle-GPU companies planning a new platform that named a cloud platform or model provider among their options90%108 of 120
Idle-GPU companies planning a new platform that named an on-premises cluster or self-managed open-source stack8%9 of 120
Companies running their own stack whose GPUs are busy half the time or less56%18 of 32
All other companies running AI in production whose GPUs are busy half the time or less83%154 of 186
September only: companies planning a new platform with no finance-reviewed ROI figure91%58 of 64

Three details in that table carry more weight than the headline. First, the planning rate is statistically indistinguishable between companies with idle GPUs and companies with busy ones, 70 versus 74 percent, which means idle capacity is not damping expansion appetite at all. Second, when idle-GPU companies plan new platforms, 90 percent look at cloud platforms or model providers and only 8 percent at infrastructure they would control themselves. Third, the companies actually running their own stacks report materially better utilisation, 56 percent idle versus 83 percent for everyone else.

The survey's own methodology note is refreshingly blunt: each month drew a different self-selected group of readers and panel respondents, so the results describe those respondents rather than every company. Utilisation figures also exclude companies that said they do not measure GPU use or do not operate GPUs. This is a survey of companies aware enough to have an answer, which if anything biases toward organisations paying attention to their infrastructure.

The independent measurement is worse

Survey self-reports have a known failure mode: people underestimate what they are not measuring. Operational data on GPU utilization from the other direction suggests the self-reports are, if anything, flattering. Cast AI's 2026 State of Kubernetes Optimization Report, which analysed operational data from tens of thousands of production clusters across major cloud providers rather than asking anyone anything, found average GPU utilisation hovering around 5 percent, with enterprises provisioning roughly 20 times the GPU capacity their workloads actively consume.

The company's co-founder and president Laurent Gil told AIwire the number surprised even him: “For something as expensive and hard to find as a GPU, I expected the number to be higher.” The report benchmarks optimised environments at around 50 percent average utilisation, which means the gap between what is and what is achievable is roughly a tenfold reduction in cost per unit of useful work.

A survey and an operational dataset measuring different populations cannot be averaged, and this article will not pretend otherwise. What they agree on is the direction and the magnitude: enormous amounts of paid-for GPU time produce nothing, GPU utilization shows the same pattern every month it is measured, and the organisations closest to the hardware, the ones running their own stacks, see it most clearly.

Why companies with idle GPUs buy more anyway

The mismatch stops looking like a contradiction once you see that companies are paying for two different things that cannot substitute for each other.

The first is capacity: GPUs the company controls or reserves, much of it lined up during the shortages of 2024 and 2025, when being caught short looked like the greater risk. The second is frontier intelligence: the top models from Anthropic, OpenAI and Google, which are closed weights, delivered per token through APIs, subscriptions or the big clouds. A company cannot load a frontier model onto the GPUs it already owns. The idle capacity and the growing AI demand sit in separate silos, and the money flows to both.

The survey data fits this reading exactly. Idle-GPU companies plan new platforms at the same rate as busy ones, and when they plan, they look at cloud platforms and model providers, not at better using what they have. Meanwhile the utilisation gap between self-managed stacks, 56 percent idle, and everyone else, 83 percent, suggests that teams who see every idle hour on their own hardware manage it better, which is consistent with the 2024-era shortage purchases having been made by organisations who then moved their actual workloads to managed frontier APIs.

Add the finance context and the picture completes: 91 percent of September's platform planners had no ROI figure their finance team had reviewed. Companies are signing the next compute contract without knowing what the current one produced.

The shortage scar explains the buying behaviour

To understand why rational companies hold idle capacity and still buy more, it helps to remember what 2024 and 2025 felt like for anyone procuring compute. GPUs carried multi-month lead times, rental prices climbed through the year, and the prevailing advice in procurement circles was to secure capacity before you needed it, because the alternative, being caught short mid-project while competitors trained and shipped, looked existential. Buying ahead of demand was the rational response to the market those years created.

Markets move faster than procurement habits. The shortages eased, frontier demand consolidated onto per-token APIs, and hardware that was once impossible to find became bookable by the hour. But the contracts signed under scarcity, reserved instances, long-term commitments, owned clusters sized for a peak that arrived smaller than feared, do not unwind themselves, and the organisations holding them now carry capacity that made sense at signing and sits idle at renewal. The survey's stability across three months, 77 to 81 percent idle, reads differently with that history in mind: it is not three months of one bad quarter, it is the steady state of capacity bought for a world that changed shape.

There is also a governance explanation worth naming. GPU procurement is capital expenditure, visible and owned by infrastructure leadership, while model API spending is operating expenditure that grows quietly inside engineering budgets. The survey's most damning number, 91 percent of platform planners with no finance-reviewed ROI figure, suggests those two ledgers are not being reconciled anywhere. A company that reviews its GPU bill at the board level and its token bill nowhere will keep buying the thing it measures and ignoring the thing it does not, and no amount of market commentary fixes that. The fix is a single report that puts both ledgers on one page, measured in the only unit that matters: cost per completed task.

What the token bill changes

The survey's own analysis points at where the next squeeze lands: model usage. As companies shift work to agents, which plan and carry out multi-step tasks, per-token charges grow even as per-token prices fall, because agents consume far more tokens per unit of finished work. Gartner has predicted inference costs per agentic workflow will rise more than fivefold through 2028. Coding agents re-read their context at every step, and the engineering response, caching, is already priced in: Anthropic charges between one-fortieth and one-tenth of normal input price for cached tokens, and two Nvidia engineers reported that in a Claude Code session, later calls reuse 85 to 97 percent of their context from cache.

Multi-agent setups multiply the bill. Vals AI found coding-agent teams, a lead agent running up to five subagents, cost 1.8 to 5.1 times as much as a single agent, with only one of four team setups improving benchmark scores by a statistically significant margin. This site has covered that economics from several angles, including the harness cost study, model routing and inference caching, and the survey's conclusion matches all of them: the controllable cost is usually on the token side, not the hardware side.

Six moves before the next platform contract

The survey's closing list is the practical core, and it is worth reproducing with commentary, because every item is something a team can start this week rather than next budget cycle.

Measure cost per completed task, not per token. Across both GPU capacity and API spending. A cheaper model that uses three times the tokens costs more per task, and only task-level measurement reveals it.

Test default models by finished-work economics. The company default model and effort level should be chosen by cost per completed task, judged on your actual workload, not on leaderboard scores or price sheets.

Send each agent only the context it needs, and cache the rest. Whether through model-provider cache APIs or your own infrastructure. This is the single highest-leverage token saving, and the 85 to 97 percent cache-reuse figure shows how much repeated context ordinary agent sessions carry.

Add a second agent only when the task splits. The Vals AI numbers say most multi-agent setups add cost without adding accuracy. Parallelism is a tool for divisible work, not a default architecture.

Move routine volume to open models on idle capacity. Classification, extraction and summarisation are exactly the workloads that do not need a frontier model and exactly the workloads that can fill idle GPUs. This is the bridge between the two silos, and it converts waste into capacity without a new contract.

Fill remaining idle hours with deferrable work. Batch inference, model evaluations, fine-tuning. Any organisation with both idle GPUs and a queue of deferred evaluation or fine-tuning work owns the fix to its own utilisation problem.

Where the numbers stop

  • The 79 percent figure describes 218 self-selected survey respondents who reported utilisation, not the market. The pooled surveys' own methodology section says month-to-month differences are statistically insignificant and group differences count only at p below 0.05.
  • The Cast AI 5 percent figure measures Kubernetes-managed GPU clusters at cloud providers, a different population from the survey's, and the two numbers measure different things: share of time idle versus share of capacity utilised. They agree in direction, not in quantity, and no average between them is meaningful.
  • The survey did not ask how respondents obtained their GPU capacity or whether they control it directly, so the capacity-versus-APIs explanation in the analysis is the author's interpretation, clearly flagged there, and repeated here as interpretation rather than finding.
  • Some idle capacity is deliberate: reserved capacity for demand spikes and older platforms awaiting replacement both depress measured utilisation. The data cannot separate deliberate slack from waste.
  • The Gartner inference-cost forecast, the Nvidia engineers' cache-reuse figures, the Anthropic cache pricing and the Vals AI multi-agent costs are cited by the survey from third-party work; this article has not independently verified those sources and carries them at the survey's valuation.
  • No figure in the survey or this article establishes what any individual reader's organisation should spend. The denominator that matters is your own utilisation data, which is why the first move is measuring it.

FAQ

How much GPU capacity is actually sitting idle?

Two independent measurements agree the waste is large. A pooled survey of 218 companies running AI in production found 79 percent say their GPUs are busy half the time or less, a figure that held between 77 and 81 percent across three monthly waves from July to September 2026. An operational analysis of tens of thousands of production Kubernetes clusters by Cast AI found average GPU utilisation around 5 percent, implying roughly 20 times more capacity provisioned than used. The two numbers measure different populations and cannot be averaged, but both point the same direction.

Why do companies with idle GPUs plan to buy more compute?

Because the idle capacity and the demand are in different silos. Frontier models from Anthropic, OpenAI and Google are closed-weight and billed per token, so growing AI demand flows to model providers and clouds rather than onto GPUs the company already controls. The survey supports this: 70 percent of idle-GPU companies plan a new platform, statistically the same as busy-GPU companies, and 90 percent of those planners named a cloud platform or model provider among their options, while only 8 percent named self-managed infrastructure. Companies running their own stacks report better utilisation, 56 percent idle versus 83 percent for others.

What should a company do before signing a new AI compute contract?

Six things, in rough order: measure the cost of each completed task across GPU and API spending; pick default models by cost per finished task rather than token price; send agents only the context they need and cache repeated context, which cuts input costs by up to 97 percent on cached tokens; add multi-agent setups only when tasks genuinely split into parallel pieces; move routine high-volume work such as classification and extraction onto open models running on the idle capacity; and fill remaining idle hours with batch inference, evaluations and fine-tuning. The goal is more finished work from compute already paid for, which is the only benchmark the next contract should be judged against.

The short version

Survey data across three months shows 79 percent of companies running AI in production reporting GPUs idle at least half the time, with 70 percent of them planning new compute platforms anyway and 91 percent of September's planners lacking any finance-reviewed ROI figure. Operational data from Cast AI puts cluster utilisation near 5 percent against an achievable 50. The mismatch is structural: closed frontier models route demand to token bills while shortage-era capacity sits unused, and only companies running their own stacks, at 56 percent idle, see and manage the gap directly. The response list is unglamorous and immediate: measure cost per task, cache context, resist reflexive multi-agent setups, and push routine work onto the GPUs you are already paying for.

Sources
Subscription Form