vLLM is 24 times faster, or 2-4 times, depending on the baseline
The 24x figure is measured against HuggingFace Transformers; quote it to a client already running decent serving and you promise more than the paper does.
In brief
- vLLM came out of the Sky Computing Lab at UC Berkeley, is built on PagedAttention, and has been a PyTorch Foundation project since May 2025.
- vLLM's server is compatible with the OpenAI API, so moving existing client code on-prem mostly means changing the endpoint address.
- The 24x and 2-4x figures are both correct but use different baselines. Both date from 2023, so clients still need to benchmark their own workloads.
In June 2023, the blog post launching vLLM claimed throughput up to 24 times higher than HuggingFace Transformers. Three months later, the academic paper on PagedAttention by Kwon et al. gave a far more modest figure: 2-4 times.
Neither number is wrong. The 24x figure uses HuggingFace Transformers as its baseline. The 2-4x figure compares against the best serving systems of the time, at the same level of latency.
For an FDE, the distinction has practical consequences. Suppose the client’s infrastructure team already runs a decent serving system and you walk into the meeting with the 24x figure: you are promising something the original paper itself does not promise. To understand vLLM, start with what it actually optimises.
PagedAttention borrows an idea from operating systems
When an LLM generates text, each request has to keep the attention keys and values for the tokens it has already processed. This memory is called the KV cache. Put simply, the more requests running concurrently, the more KV cache has to be held, so the way this memory is managed determines how many requests a server can accept at once.
vLLM’s core feature is PagedAttention, an attention algorithm inspired by virtual memory and paging in operating systems. An operating system splits memory into pages, and PagedAttention manages the KV cache in pages in the same way. vLLM combines this with continuous batching.
That is why vLLM’s benefit is clearest when many users are active at once. A demo with a single person typing will not show much.
When does an FDE need vLLM?
The typical case is a client that wants to run open-weight models inside its own infrastructure. vLLM then addresses three concerns at once.
The first is hardware. Beyond NVIDIA, vLLM officially supports AMD, Google TPU, AWS Neuron, Intel CPU/XPU/HPU and ARM. Accelerators such as IBM Spyre and Huawei Ascend are integrated through a plugin system. On-prem clients usually have a mixed fleet of hardware already, and you rarely get to choose from scratch.
The second is a model too large for a single machine. The official repository supports tensor, pipeline, data, expert and context parallelism for distributed inference.
The third is legal. vLLM is licensed under Apache-2.0, and since May 2025 it has been a PyTorch Foundation project with neutral governance. Both make it easier for the client’s procurement and legal teams to sign off.
A minimal example: change one address, keep the code
Imagine the client already has a prototype calling OpenAI through the official SDK, but data can no longer leave the internal network. vLLM has an OpenAI-compatible API server, along with the Anthropic Messages API and gRPC, so the client-side work can come down to a single line. In the snippet below, the comment reads “Illustration: point the client at the vLLM server inside the client’s network”, and the placeholders stand for the internal vLLM endpoint address and an internal key:
from openai import OpenAI
# Illustration: point the client at the vLLM server inside the customer's network
client = OpenAI(base_url="INTERNAL_VLLM_ENDPOINT_URL", api_key="INTERNAL_KEY")
Prompt logic, result handling and retry mechanisms all stay the same. This is the strongest argument for winning over the client’s application team: they do not have to rewrite anything. The docs add one small caveat: the Completions and Chat APIs are only for text-generation models, so do not plan to call an embedding model through the same route.
Two day-one traps
The first trap is the chat template. The docs are blunt: without a chat template, the server cannot handle chat, and every chat request returns an error. Open-weight models downloaded from unfamiliar sources, or internal fine-tunes, can easily lack one. Check before the application team connects, rather than waiting for them to report errors.
The second trap only appears in air-gapped environments. By default, FastAPI’s /docs page needs an internet connection, so on an isolated network it does not render. vLLM has a flag to serve the page in offline mode. Turn it on in advance and you will not lose an afternoon explaining why “the documentation page is blank”.
What to learn before taking the job
Learn the concepts first, the tool second. Understand what the KV cache is and why the way it is managed affects how many requests can be served at once. Then stand up a vLLM server yourself, point an OpenAI client at it and measure throughput as you gradually increase the number of parallel requests.
If you are applying for FDE roles, look for job descriptions that mention “on-prem”, “air-gapped” or “open-weight models”. On a CV, a line such as “migrated an application from the OpenAI API to on-prem vLLM with no changes to client logic, resolved chat template errors” says far more than “experience with LLMs”.
When quoting benchmarks to a client, always state the baseline, and remember that the 2-4x figure was measured in 2023 on the popular models of the time. The best recommendation you can make is to benchmark on the client’s own workload and hardware before promising any number.
Was this article useful?
Thanks for the feedback!
6 sources
- Welcome to vLLM
- Online Serving
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention · 2023-06-20
- Efficient Memory Management for Large Language Model Serving with PagedAttention · 2023-09-12
- PyTorch Foundation Welcomes vLLM as a Hosted Project · 2025-05-06
- vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs