Two provider tabs are open, a coding agent’s transcript just crossed 400K tokens, and the deadline is an hour out. One tab quotes a flat per-token rate. The other quotes something lower at the top of the page with a footnote about pricing above 128K or 200K tokens. The tiered quote might beat the flat one, or it might triple once the window fills to 1M. You can’t tell from the landing page alone.
The spread exists before context even enters the picture
Pull the same model across every provider we track and the cheapest-to-priciest gap at a single, short context length is already large. qwen3-235b-a22b and llama-3-1-405b both show spreads north of 20x, cheapest provider to priciest, as of this writing. That’s the floor problem. Long context stacks a second variable on top: the tier a provider advertises is almost always the short-context tier, and the number that applies once your prompt crosses whatever threshold they set is a different line on a different page.
The Anthropic trap runs the other direction
Anthropic used to charge a premium for prompts over 200K tokens on its longest-context models. That surcharge is gone now. If you built a cost model, a spreadsheet, or an internal router around that old tier structure, it’s overcounting Anthropic on every long-context call, not just the marginal token past the old threshold. Stale long-context math doesn’t just make providers who still tier look artificially cheap by comparison, it makes providers who dropped the tier look artificially expensive. Check the live model page before trusting last quarter’s assumption either way.
Not every provider followed Anthropic’s move. Several still price prompts above a threshold at a materially higher rate than the same model’s short-context calls. Filling 1M tokens against a quote you sourced from a 4K-token benchmark compares the wrong tier entirely.
Self-hosting swaps the pricing trap for a memory one
Renting a GPU sidesteps tiered per-token pricing completely: you pay for GPU-hours, and a longer prompt costs you time, not a rate change. It does not sidestep the memory bill.
KV cache scales with sequence length, layer count, and attention heads, not with total parameter count. The rough rule of ~1.1GB per billion params at FP8, plus roughly 25% headroom for KV cache, holds fine at a working context of a few thousand tokens. Push that same model to a 1M-token window and the 25% assumption breaks. KV cache alone can add hundreds of GB on top of the weights, illustratively, for architectures with many layers and heads. A 1M-token session on a large model like kimi-k2-thinking or deepseek-v3-2 can need multi-GPU room just for cache, regardless of whether the active-parameter compute bill is modest. Check h200 or b200 pricing at the memory tier you’d need before assuming a single-card rental sized off the model’s file size covers a long-context deployment.
I’ve watched a team size a box off the model’s disk footprint, get comfortably under one GPU’s memory, then OOM at request 40 when a user pasted a full log file and the live session crossed 300K tokens. The fix was a mid-incident redeploy onto a bigger box. Hours lost on a problem that a KV-cache-at-p95 calculation would have caught before launch. gb300 headroom exists for exactly this case, but only if you check it before you need it.
Compaction beats the whole comparison
None of the tier math or the GPU sizing matters if the workload doesn’t need the full window. Most long agent sessions don’t need 1M raw tokens of history, they need the last handful of tool outputs and a compressed summary of everything before that. Compaction, rewriting old turns into a shorter summary before the next call, trades a little recompute for a much smaller live context on every subsequent turn. On per-token pricing that recompute pays for itself on the very next call. On a rented GPU it’s the difference between one card and four.
The general economics of renting versus per-token pricing still apply here. They just move faster than usual, because both sides of the equation shift with context length, not just with model choice. Run the numbers again this week rather than trusting last month’s comparison in the calculator.
The rule to apply today
If your sessions sustain past roughly 200K tokens on a majority of calls, check the model’s live page for a tier ladder above that mark before extrapolating from the headline rate to a 1M-token bill. If they clear that mark and you’re self-hosting, size the box for KV cache at your p95 context length, not average, and recheck against h100-sxm pricing at that footprint before renting. If your transcripts are long but the load-bearing information inside them rarely exceeds a few hundred K tokens, fix that with compaction before comparing providers at all, because no tier ladder and no GPU rental beats not paying for tokens you didn’t need to keep. What none of this can tell you is how much of your own 1M-token prompt is genuinely load-bearing versus stale scaffolding an agent never rereads. That number lives only in your own traces.