A team ships a support bot on Ollama running GLM-4.5 Air on a single RTX 4090. It works great in the demo: one user, one request, sub-second first token. Three weeks later it’s in production with 30 concurrent support agents hitting it during a shift change, and p95 latency goes from under a second to 12+ seconds. The fix the team reaches for is more GPUs behind a load balancer. The actual problem is the serving engine, not the hardware.
Why Ollama chokes under concurrency
Ollama wraps llama.cpp. Its batching has improved, but the default request handling is still built around serving one or a handful of sequences well, not scheduling hundreds of in-flight requests with different prompt lengths and generation lengths. Each new request effectively waits its turn or shares a small batch slot. Throughput per GPU flattens fast as concurrency rises, because the scheduler isn’t packing requests into the GPU’s actual compute headroom.
vLLM’s whole design point is continuous batching with PagedAttention: KV cache lives in fixed-size blocks the scheduler can add to and evict from mid-generation, so a GPU running vLLM can have 50 sequences in flight, each at a different stage of decoding, all packed into the same forward pass. That’s the difference between a card that’s busy and a card that’s busy with 3 users while 27 wait in a queue.
You can measure this on your own hardware in an afternoon: run the same model on vLLM and Ollama on identical H100 SXM capacity, ramp concurrent requests from 1 to 50, and plot latency. Ollama’s curve bends sharply upward well before 50; vLLM’s stays close to flat until it saturates the card’s actual FLOPs. Community-tier rental capacity on cards like MI300X runs 70%+ cheaper than secure tier as of this writing, which makes this comparison cheap enough to run yourself before you commit a production deploy on either engine.
When Ollama is the right call
None of this is a knock on Ollama for the job it’s built for: single-user local inference, laptop and workstation prototyping, quick evals of a new checkpoint before you commit rack budget. If your traffic is one developer or a handful of internal users who tolerate queueing, the concurrency gap never bites, and Ollama’s model pulls and GGUF quant management save real setup time over hand-rolling a vLLM deployment. A Granite 4.1 8B or Phi-4 demo on a single card doesn’t need a scheduler built to handle hundreds of simultaneous sequences.
The trap is treating “works in the demo” as evidence it holds at 10x the load. Concurrency in Ollama doesn’t degrade gracefully. It degrades in queue depth, and queue depth stays invisible until a dashboard shows a p95 that ate the SLA.
The fork-only wrinkle nobody warns you about
The 2026 serving stack is more fragmented than it was two years ago, because new architectures now ship with vendor forks weeks before upstream vLLM or Ollama support lands. DeepSeek V4 Flash’s MoE routing needed a forked vLLM build before the merge landed, and teams pinned to mainline releases were stuck running an older, architecture-compatible checkpoint or waiting it out. SGLang has in some cycles picked up new MoE architectures faster than vLLM mainline. Check both projects’ support matrices before you commit a production deploy to a brand-new model release, not just the one you’ve already standardized on.
Speculative decoding isn’t a free win
MTP (multi-token prediction) speculative decoding is one of the better throughput levers in current serving stacks, but the gain is hardware-dependent in a way that catches people who tune once and assume it transfers. On data-center GPUs with strong tensor core throughput, MTP’s extra draft-and-verify passes are cheap relative to the tokens they save, close to a straight win. On Apple Metal backends the same technique can come out negative: the draft pass competes for the same limited compute the main model needs, and Metal’s per-forward-pass kernel overhead doesn’t amortize the way CUDA’s does. If you benchmark speculative decoding wins on a Mac Studio during prototyping and assume they hold once you move to an H200 in prod, don’t. Rerun the benchmark on the actual target hardware; the two curves can point in opposite directions.
The decision rule
Estimate peak concurrent requests per GPU, not average load. If that number clears somewhere around 8-10 simultaneous requests hitting one card during normal traffic, Ollama’s scheduler is already leaving throughput on the table, and you should be moving to vLLM or SGLang before the number grows further. Below that, on a single dev box or workstation, Ollama’s simplicity is worth keeping. What this guide can’t tell you is your own queueing tolerance: a 12-second p95 might be fine for an internal tool and fatal for a customer-facing chat widget, and only your own traffic logs know which one you’re running.