For the first time, three generations of NVIDIA’s flagship inference hardware are rentable side by side, and the pricing spread between them changes weekly (live floors). The right choice depends on what constrains your workload: memory, bandwidth, or compute.
Where the three differ
| H100 SXM | H200 | B200 | |
|---|---|---|---|
| VRAM | 80 GB HBM3 | 141 GB HBM3e | 180 GB HBM3e |
| Memory bandwidth | 3.35 TB/s | 4.8 TB/s | 8 TB/s |
| Architecture | Hopper | Hopper | Blackwell |
| Native FP4 | no | no | yes |
The line that matters most for inference is memory bandwidth. LLM decoding is bandwidth-bound: every generated token re-reads the model weights. The H200’s 43% bandwidth bump over the H100 translates into a comparable throughput bump. Check whether the current H200 premium is below that before defaulting to H100s.
Choose by constraint
Constrained by VRAM (the model barely fits)? The H200’s 141 GB is the sweet spot. A 70B model at FP8 that needs tensor-parallelism across two H100s fits on one H200, and one GPU always beats two on cost, latency and failure modes. This is the most common upgrade that pays for itself.
Constrained by throughput on a fitting model? Compare $/hr ÷ bandwidth. On that ratio the H100 is often still the value king when its rental discount is steep; let today’s floors settle it rather than the spec sheet.
Running MoE giants or FP4-quantized models? The B200 is a different class: 180 GB, about two-thirds more bandwidth than an H200, and native FP4 that roughly doubles effective capacity again. DeepSeek-class MoE models (671B total) that need eight H100s fit on far fewer B200s. But rental premiums are still early-adopter priced. The B200 page tracks the floor daily, and it has been falling.
Don’t forget the off-brand answer
If your model fits in 192 GB and your stack supports ROCm, the MI300X frequently undercuts all three NVIDIA cards per hour while offering the most VRAM per GPU of the Hopper generation’s rivals. vLLM support is mature; the friction is real but shrinking.
Rules of thumb
- 7B-32B models: none of the above. An L40S or even an RTX 5090 serves these at a fraction of the price.
- 70B dense at FP8: one H200 beats two H100s. Compare against MI300X.
- 100B+ MoE: B200 nodes if the premium has compressed; otherwise H200 clusters.
- Long-context anything: KV cache eats VRAM linearly with context, so the H200/B200’s extra memory is worth more than the spec sheet suggests.
These premiums drift week to week. Bookmark the per-GPU pages for the current spread, and the trends page to see which direction each one is moving before you commit to a fleet.