Every team serving an open-weight model eventually asks: we’re paying per token, should we just rent the GPU? A short formula settles it. Here it is, with the traps.
The formula
Self-hosted cost per million output tokens:
$/1M = (GPU count × $/hr) ÷ (aggregate tok/s × 3600 × utilization) × 1,000,000Worked example: Llama 3.3 70B at FP8 needs ~96 GB of VRAM, so one 141 GB H200 fits it. Take an illustrative $3.30/hr on-demand rate (the live H200 price will differ by the time you read this) and a well-batched vLLM deployment doing ~650 aggregate tok/s at 50% utilization:
(1 × 3.30) ÷ (650 × 3600 × 0.5) × 1,000,000 ≈ $2.82 per 1M tokensCompare that against the cheapest API price on the model’s page (live prices here) and you have your answer for today. Our calculator runs this with live GPU floors.
The three variables that move the answer
1. Utilization. The GPU bills 24/7; your traffic doesn’t arrive 24/7. A chatbot with strong daytime peaks might average 30% utilization; an offline batch pipeline can hit 95%. Utilization divides directly into your cost: the same deployment at 25% utilization costs 4× as much per token as at 100%. This single number flips most decisions.
2. Aggregate throughput, not per-request speed. A single request might stream 40 tok/s, but a batched vLLM server answers dozens of requests concurrently; the aggregate is what earns back the rent. Aggregate throughput scales with the model’s active parameters. A MoE model like GPT-OSS 120B (5B active) batches like a small model while answering like a big one, which is why MoE models are disproportionately cheap to self-host.
3. The API floor keeps moving. Open-model API prices have compressed relentlessly; providers on our model pages reprice weekly. A breakeven that cleared 3× margin in January can be underwater by June. Put a quarterly reminder on the math.
Costs the formula hides
- Input tokens are not free. The formula above prices output; a RAG workload pushing 20:1 input:output ratios shifts the math (APIs charge input at 2-5× less than output, but they do charge).
- Engineering time. Someone owns quantization choices, CUDA driver mishaps, autoscaling, and the 2 a.m. page. At small scale this dwarfs the compute delta.
- Redundancy. One GPU is not a production deployment. Two is your real minimum, which doubles the rent until traffic fills both.
- Egress and storage are rounding errors for LLM serving, but check anyway on marketplace hosts.
The honest decision rule
- Under ~200M output tokens/month: use the API. The engineering overhead swamps any savings.
- Steady, high-utilization workloads past ~500M tokens/month on a model whose API price is above your computed self-host cost: rent, starting with one node of the cheapest fitting GPU (floors on our GPU board).
- Bursty or latency-critical traffic: hybrid. Serve the base load on rented hardware, overflow to the API.
Run your own numbers in the breakeven calculator. It uses today’s live prices on both sides of the equation.