Neither a successful single-user demo nor a serving engine’s reputation proves that it will meet your production target. This guide previously presented invented latency results and a universal eight-to-ten-request migration threshold. Those claims are withdrawn; no comparative benchmark was performed for this article.
What the projects actually document
The vLLM project documents continuous batching and PagedAttention among its serving features. These describe implementation capabilities, not a guarantee of flat latency as traffic rises.
Ollama’s FAQ documents parallel request processing, queuing and concurrency controls. Parallel contexts increase memory requirements. It is therefore incorrect to describe Ollama as inherently unable to handle concurrent production requests. Defaults and supported behavior can change, so record the installed version and settings.
Run a comparison that can be reproduced
- Select a checkpoint supported by both engines. Record model revision, tokenizer, quantization, context limit, chat template and generation settings. If formats differ, report that difference and check answer quality.
- Use the same GPU, memory allocation and host configuration. Separate cold model-loading time from warm serving measurements.
- Replay a fixed set of prompts with realistic input and output lengths. Include long-context and burst cases rather than only short prompts.
- Increase offered load in steps. Test both fixed concurrency and arrival rates; a closed-loop client can conceal queues by waiting for every response before sending another.
- Measure time to first token, inter-token latency, completion latency, completed requests per second, errors, queue depth and memory. Report p50, p95 and p99 with sample counts and test duration.
- Repeat runs and save commands, configurations and raw results. Compare successful responses that meet your quality and latency requirements.
Do not select the winner from total generated tokens alone. A system can produce more tokens while missing your response deadline or rejecting more requests. Also distinguish aggregate throughput from each user’s decoding speed.
Turn results into a deployment choice
Choose the smallest configuration that meets your defined peak-load target with an explicit operating margin. Price the whole configuration, including replicas, idle time and maintenance, against the GPU catalog. Keep a rollback path and observe real queues after launch.
The vLLM overview and Ollama overview are starting points for setup. Confirm current model and hardware support in the projects’ documentation before spending on a test.
Reviewed November 18, 2026. This is a test protocol. It does not claim either engine wins at a particular concurrency, nor that a model fits a particular GPU.