GPU Cost per 1M Tokens: Why the Cheapest $/hr GPU Isn't the Cheapest to Run
When you shop for cloud GPUs, every comparison site shows price per hour. It's obvious and easy to scan. But if you're running an LLM — serving, not training — the number that actually hits your bill is cost per token, and it behaves very differently from the hourly rate.
Here's the counterintuitive finding from the live pricing data running on gpuprice.in today: the cheapest GPU by the hour is also the most expensive GPU by the token.
📊 Compute it for your exact model & GPU
Pick a workload, pick a GPU, get live $/1M-token cost from the live feed — not a ballpark.
Workload Cost Calculator →The trap in one table
This is the live cheapest on-demand rate per GPU (from the gpuprice.in feed, up-to-date daily) crossed with a realistic batched-serving throughput for Llama 3.1 70B, then converted to dollars per 1M tokens:
| GPU | $/hr (live) | tokens/sec (70B, batched) | $/1M tokens | Verdict |
|---|---|---|---|---|
| RTX 5090 | $0.58 | — (32GB can't hold 70B) | N/A | Doesn't fit |
| H100 | $2.50 | 600 | $1.16 | Fast & cheap per token |
| H200 | $2.80 | 600 | $1.30 | Same speed, more VRAM |
| B200 | $4.50 | 900 | $1.39 | Fastest, premium price |
| A100 | $1.67 | 250 | $1.86 | Cheapest $/hr, priciest $/token |
A100 is the cheapest per hour ($1.67) and yet the most expensive per token ($1.86). The H100 costs 50% more per hour, but it's 38% cheaper per token because it serves more than twice as many tokens per second. If your GPU is busy, the $/hr sticker price is actively misleading.
Small models flip everything
The same logic, applied to a small model like Llama 3.1 8B, makes the ranking even more lopsided — because small models saturate GPUs and every card serves thousands of tokens per second batched:
| GPU | $/hr (live) | tokens/sec (8B, batched) | $/1M tokens |
|---|---|---|---|
| RTX 5090 | $0.58 | 6,000 | $0.03 |
| H100 | $2.50 | 8,000 | $0.09 |
| H200 | $2.80 | 8,000 | $0.10 |
| A100 | $1.67 | 4,500 | $0.10 |
| B200 | $4.50 | 12,000 | $0.10 |
For an 8B model, the RTX 5090 at ~$0.03/1M tokens crushes every data-center card. You don't need a $2.50/hr H100 for a 32GB model — you're renting far more compute than the model can use. This is where the budget-minded player wins big.
Why the per-hour price misleads
The formula behind every number above is straightforward:
| $/1M tokens = ( $/hr ÷ ( tokens/sec × 3,600 ) ) × 1,000,000 |
Three inputs drive it, and only one of them is the sticker price:
- $/hr — the one everyone compares.
- Tokens/sec — how fast the GPU actually serves your model while batched with real user traffic. This is model- and workload-specific, not a spec sheet number.
- Utilization — the same formula only pays off if the GPU is actually busy. Idle hours are pure waste at any rate.
GPU rental vs. inference API, by the token
Here's where renting a GPU gets interesting. Hosted inference APIs typically charge $2–$3 per 1M tokens for a model like Llama-70B. A rented H100 serving the same model batched costs $1.16 per 1M tokens. That's roughly half to a third the API price — but only while the GPU stays busy.
Renting wins when your workload is steady and you can keep the GPU busy. An API wins on spikes, low traffic, and zero-ops convenience. Match the pricing model to your traffic shape, not your current month's bill.
Cheapest GPU per token — decision guide
| Your workload | Cheapest GPU per token | Why |
|---|---|---|
| Serving 70B+ model, busy & steady | H100 | Best $/token among 80GB cards — A100 is slower, not cheaper per token |
| Long-context 70B+ that won't fit in 80GB | H200 / B200 | Only 141GB+ cards fit; pay the VRAM premium |
| Small models (≤32GB): chat apps, pipelines | RTX 5090 | ~$0.03/1M, a fifth the H100's per-token cost |
| Sporadic / spikey traffic | Any API | Don't rent idle compute; per-token API beats wasted hours |
| Training / fine-tuning (not tokens) | Cheapest $/hr fit | Token math doesn't apply — cost = $/hr × wall-clock hours |
Methodology — how these numbers are grounded
Prices are real: the live cheapest on-demand $/hr per GPU straight from the gpuprice.in feed (source: Parallel.ai), dated daily. Throughput is the one honest caveat: tokens/sec is a batched-serving estimate for that model on that GPU under multi-user load — it is a methodology default and varies with framework, batching, quantization and model shape. It is labeled as an adjustable default in our calculator, never claimed to be a measured feed figure. Change the throughput to your own benchmark and the rankings can shift — that's exactly what the calculator is for.
📊 Get your real $/1M-token number
Enter your actual throughput, or start from the live defaults, and see the cost for every GPU that has enough VRAM for your model.
Open the Calculator →Frequently asked questions
How is the cost per 1M tokens calculated for a rented GPU?
It's the hourly rate divided by how many tokens the GPU serves per second, scaled up: ($/hr ÷ (tokens/sec × 3,600)) × 1,000,000. The tokens-per-second figure is a batched-serving estimate for that model on that GPU — a methodology default you can adjust, not a live feed price.
Why isn't the cheapest $/hr GPU the cheapest $/token GPU?
Because a slower GPU rents more hours to produce the same tokens. For serving Llama-70B batched today, the A100 is cheapest per hour ($1.67) but most expensive per 1M tokens ($1.86), while the H100 is $1.16.
Which GPU is cheapest to run for small models?
The RTX 5090 — about $0.03 per 1M tokens for 8B-class work, because it's cheap per hour and still serves thousands of tokens per second batched. It just can't fit 70B+ models in 32GB.
Does the RTX 5090 appear in the per-token comparison?
Only for models small enough to fit its 32GB. The comparison is VRAM-gated: a GPU only appears for a model if its memory is large enough.
When is renting cheaper than an inference API?
When the GPU stays busy. Renting an H100 to serve Llama-70B costs ~$1.16/1M tokens versus $2–$3/1M on a hosted API — roughly half to a third, but only at steady utilization. On spikes or low traffic, the API's per-token price wins.
Live prices August 8, 2026 (Parallel.ai feed). Throughput = batched-serving methodology defaults, adjustable in the calculator. See the full methodology on the calculator page.