# vLLM is 24 times faster, or 2-4 times, depending on the baseline

> The 24x figure is measured against HuggingFace Transformers; quote it to a client already running decent serving and you promise more than the paper does.

Original: https://fdetimes.net/en/tools/vllm-open-weight-serving-on-prem/

In June 2023, the blog post launching vLLM claimed throughput up to 24 times higher than HuggingFace Transformers. Three months later, the academic paper on PagedAttention by Kwon et al. gave a far more modest figure: 2-4 times.

Neither number is wrong. The 24x figure uses HuggingFace Transformers as its baseline. The 2-4x figure compares against the best serving systems of the time, at the same level of latency.

For an FDE, the distinction has practical consequences. Suppose the client's infrastructure team already runs a decent serving system and you walk into the meeting with the 24x figure: you are promising something the original paper itself does not promise. To understand vLLM, start with what it actually optimises.

## PagedAttention borrows an idea from operating systems

When an LLM generates text, each request has to keep the attention keys and values for the tokens it has already processed. This memory is called the KV cache. Put simply, the more requests running concurrently, the more KV cache has to be held, so the way this memory is managed determines how many requests a server can accept at once.

vLLM's core feature is PagedAttention, an attention algorithm inspired by virtual memory and paging in operating systems. An operating system splits memory into pages, and PagedAttention manages the KV cache in pages in the same way. vLLM combines this with continuous batching.

That is why vLLM's benefit is clearest when many users are active at once. A demo with a single person typing will not show much.

## When does an FDE need vLLM?

The typical case is a client that wants to run open-weight models inside its own infrastructure. vLLM then addresses three concerns at once.

The first is hardware. Beyond NVIDIA, vLLM officially supports AMD, Google TPU, AWS Neuron, Intel CPU/XPU/HPU and ARM. Accelerators such as IBM Spyre and Huawei Ascend are integrated through a plugin system. On-prem clients usually have a mixed fleet of hardware already, and you rarely get to choose from scratch.

The second is a model too large for a single machine. The official repository supports tensor, pipeline, data, expert and context parallelism for distributed inference.

The third is legal. vLLM is licensed under Apache-2.0, and since May 2025 it has been a PyTorch Foundation project with neutral governance. Both make it easier for the client's procurement and legal teams to sign off.

## A minimal example: change one address, keep the code

Imagine the client already has a prototype calling OpenAI through the official SDK, but data can no longer leave the internal network. vLLM has an OpenAI-compatible API server, along with the Anthropic Messages API and gRPC, so the client-side work can come down to a single line. In the snippet below, the comment reads "Illustration: point the client at the vLLM server inside the client's network", and the placeholders stand for the internal vLLM endpoint address and an internal key:

```python
from openai import OpenAI

# Illustration: point the client at the vLLM server inside the customer's network
client = OpenAI(base_url="INTERNAL_VLLM_ENDPOINT_URL", api_key="INTERNAL_KEY")
```

Prompt logic, result handling and retry mechanisms all stay the same. This is the strongest argument for winning over the client's application team: they do not have to rewrite anything. The docs add one small caveat: the Completions and Chat APIs are only for text-generation models, so do not plan to call an embedding model through the same route.

**Key point:** At a client site, vLLM's greatest value is often not speed but the fact that existing code does not need rewriting when it moves on-prem.

## Two day-one traps

The first trap is the chat template. The docs are blunt: without a chat template, the server cannot handle chat, and every chat request returns an error. Open-weight models downloaded from unfamiliar sources, or internal fine-tunes, can easily lack one. Check before the application team connects, rather than waiting for them to report errors.

The second trap only appears in air-gapped environments. By default, FastAPI's `/docs` page needs an internet connection, so on an isolated network it does not render. vLLM has a flag to serve the page in offline mode. Turn it on in advance and you will not lose an afternoon explaining why "the documentation page is blank".

## What to learn before taking the job

Learn the concepts first, the tool second. Understand what the KV cache is and why the way it is managed affects how many requests can be served at once. Then stand up a vLLM server yourself, point an OpenAI client at it and measure throughput as you gradually increase the number of parallel requests.

If you are applying for FDE roles, look for job descriptions that mention "on-prem", "air-gapped" or "open-weight models". On a CV, a line such as "migrated an application from the OpenAI API to on-prem vLLM with no changes to client logic, resolved chat template errors" says far more than "experience with LLMs".

When quoting benchmarks to a client, always state the baseline, and remember that the 2-4x figure was measured in 2023 on the popular models of the time. The best recommendation you can make is to benchmark on the client's own workload and hardware before promising any number.

**Try this week:**

- Run vLLM with a small open-weight model, then point a script that uses the OpenAI SDK at that endpoint, changing only the address.
- Deliberately pick a model with no chat template, send a chat request and read the error message so you recognise the symptom.
- Read the evaluation section of the PagedAttention paper and note the baseline before quoting any figure to a client.

## Sources

- [Welcome to vLLM](https://docs.vllm.ai/en/latest/)

- [Online Serving](https://docs.vllm.ai/en/latest/serving/online_serving/)

- [vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention](https://vllm.ai/blog/2023-06-20-vllm)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)

- [PyTorch Foundation Welcomes vLLM as a Hosted Project](https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/)

- [vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs](https://github.com/vllm-project/vllm)
