Why an idle GPU wastes energy: the concurrency ladder

Look, when people want to cut the energy their AI uses, they reach for a smaller model or a newer card. Both work. But on the same model and the same cards there is a lever that moves the number further than a change of GPU generation does, and in the deployments I see it is the one nobody has looked at: how many requests the server is handling at once. So here is the exhibit: Qwen 3 32B on 2 × H100, same weights, same prompts, measured by the ML.ENERGY Leaderboard at nine different loads. At a running batch of 8 it costs 452 Wh of GPU energy per million output tokens. At 256 it costs 63.5 Wh. That is 7.1× on hardware nobody touched.

The ladder

The cap is how many requests the server was allowed to hold; the running batch is how many it actually held, on average, while it was decoding. Energy per token falls the whole way to the highlighted row. Power rises, but only 39 percent, from 850 W to 1,178 W across both cards. Tokens per second rise 9.9×, from 522 to 5,155. Median inter-token latency goes from 15 ms to 44 ms, and nobody reading an answer will notice either. Energy per token is power divided by throughput, so when throughput moves 9.9× and power moves 39 percent, energy per token moves 7.1×.

Why the under-used card costs more

A model writes one token at a time, and to write each one it has to read its whole set of weights out of the cards' memory. In plain terms, for a dense model like this one, that read is the expensive part of every step, and it costs about the same whether the step produces one token for one user or 256 tokens for 256 users. The card is a lecturer. The lecture costs the same to give to 8 people as to 256; what changes is the cost per head. See, that is why the power column barely moves while the tokens column runs away: at a batch of 8 the cards were already doing the heavy work, for very few people.

One caveat on the word idle. The benchmark never measured an empty card: every row holds at least 8 conversations, with a queue behind them. A departmental assistant with two or three requests in flight, the common case in the deployments I see, sits above the top row. Idle here means under-used, and if anything the ladder understates what under-used costs.

Most of the gain comes early

You do not need 256 requests in flight to get most of it. By a running batch of 64 the same cards read 102.0 Wh/M, 4.4× down from the top row, at 22 ms median latency and 974 W. That is 90 percent of the whole drop, at a load one busy business unit can generate. The rest of the ladder buys the remainder, and buys it more slowly.

Where it turns

More is not always better. At a cap of 384 the two cards read 64.6 Wh/M, a little above the best row, at 55 ms median latency, and the slowest token in a hundred now waits 240 ms against 148 ms one row up. The server was allowed 384 and held 340, which I read as the cards running out of memory for the conversations in flight: the engine starts juggling them, the users at the back feel it, and energy per token stops improving. The index takes the lowest-energy row at or under the leaderboard's 500 ms ceiling, the 256 row here, and the curve past it is why an index row is a floor and not a target.

A small model on one card does the same

Llama 3.1 8B Instruct on 1 × H100: 126 to 19.5 Wh/M, 6.5× on one card, power up 41 percent, throughput up 9.2×. 87 percent of the drop is in by a batch of 64, and past the best row, at a cap of 768, it turns the same way, holding 691 of the 768 it was allowed. A much smaller model, one card instead of two, the same shape. In my view the shape is the finding: it is a property of how these servers decode, not of one model, and every model's own ladder is on its page.

What it does to the bill

Electricity is the small part of this and I will not pretend otherwise. The number that moves a budget is GPU-hours, and the ladder is a GPU-hours table in disguise. At the top row a million output tokens takes 1.06 GPU-hours on these two cards; at the best row, 0.11. Same cards, same model, 9.9× the tokens out of every hour of silicon you rent or own. Turn it round for the facility: one megawatt-hour of GPU energy serves 2.2 billion output tokens at the top of the ladder and 15.8 billion at the index row, and a site with a fixed power envelope, which is every site, only gets to choose which of those two it is.

What to do with it

• Measure your running batch before you buy anything. vLLM reports it as num_requests_running; average it over a busy hour. If it is in single figures you are at the top of the ladder, and no change of card on the comparison page moves the number as far as filling the cards you have.
• Consolidate. Fewer, busier cards. Three teams each running their own copy of the same model on their own pair of cards is the usual arrangement I see; one shared deployment carrying all three sits rows further down this table, and it costs nothing in hardware.
• Ask the vendor for Wh per million output tokens at your running batch, not theirs, and put your own [N] into the RFP clause on the why page. A figure measured at 256 in flight tells you nothing about a service that will run at 8.
• Do not chase the bottom of the table. Past the best row the users wait and the energy does not fall. Pick the row on the model page that matches the latency you promised, and hold the vendor to it.

I think this is the table to take into the capacity meeting, because it is the lever you already hold. The full ladders, with the definitions, are on the index.