vLLM Local Setup: High-Throughput LLM Serving Guide

TL;DR

# Install vLLM (requires CUDA 12.1+ and Python 3.9+)
pip install vllm

# Serve a model with OpenAI-compatible API
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

# Test the endpoint
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'

# Docker deployment
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct

Caution: vLLM requires a Hugging Face account with accepted model licenses for gated models like Llama. Set HF_TOKEN in your environment before serving. Never expose the API port to untrusted networks without authentication – vLLM has no built-in auth layer.

What Is vLLM

vLLM is a high-throughput inference engine for large language models, originally developed at UC Berkeley. It is designed for one thing: serving LLMs to multiple concurrent users as fast as the hardware allows.

The core innovation is PagedAttention, a memory management technique borrowed from operating system virtual memory concepts. Traditional inference engines allocate a contiguous block of GPU memory for each request’s key-value (KV) cache. This wastes memory because the actual sequence length is unknown at allocation time. PagedAttention breaks the KV cache into fixed-size pages and allocates them on demand, reducing GPU memory waste by 60-80% in typical workloads.

The second key feature is continuous batching. Standard batching waits for a batch of requests to arrive, processes them together, and returns all results. Continuous batching dynamically adds new requests to an in-progress batch as earlier requests finish generating tokens. This eliminates idle GPU cycles between batches and keeps throughput high under variable load.

These two features combined make vLLM 2-4x faster than naive inference implementations when handling multiple concurrent requests.

When to Use vLLM vs Ollama

This is the most common question for anyone running local LLMs. The answer depends on your use case.

Ollama is built for simplicity. One command pulls and runs a model. It handles quantization, model management, and provides a clean CLI. It is the right choice for single-user desktop use, experimentation, and quick prototyping.

vLLM is built for throughput. It assumes you have a dedicated GPU server and need to serve multiple users or applications simultaneously. It does not manage model downloads or quantization – you bring your own models from Hugging Face.

If you are building a shared inference service for a team or integrating LLMs into production applications, vLLM is the better foundation. If you are running models on your workstation for personal use, Ollama is simpler and more practical.

Installation on Linux

Prerequisites

• NVIDIA GPU with CUDA 12.1 or later (check with nvidia-smi)
• Python 3.9-3.12
• At least 24GB VRAM for 7-8B parameter models at FP16; 48GB+ for 13B+
• A Hugging Face account and token for gated models

pip Install

# Create a virtual environment (recommended)
python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate

# Install vLLM
pip install vllm

# Verify installation
python -c "import vllm; print(vllm.__version__)"

The pip install pulls in PyTorch with CUDA support automatically. On a fresh system, expect the download to be 2-5GB.

From Source (for latest features or custom CUDA versions)

git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

Building from source compiles custom CUDA kernels and takes 10-20 minutes depending on hardware.

Set Up Hugging Face Authentication

# Install the HF CLI
pip install huggingface-hub

# Login (saves token to ~/.cache/huggingface/token)
huggingface-cli login

# Or set the token directly
export HF_TOKEN="hf_your_token_here"

Running Models with vLLM

Basic Serving

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000

This downloads the model from Hugging Face (first run only), loads it onto the GPU, and starts the OpenAI-compatible API server. The model stays resident in VRAM until the process is stopped.

Key Configuration Flags

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \        # Split across 2 GPUs
  --max-model-len 8192 \            # Limit context length (saves VRAM)
  --gpu-memory-utilization 0.90 \   # Use 90% of available VRAM
  --max-num-seqs 32 \               # Max concurrent sequences
  --quantization awq \              # Use AWQ-quantized model
  --dtype float16                   # Force FP16 (default on most GPUs)

Common Configuration Combinations

Single GPU, 24GB VRAM (RTX 4090 / A5000):

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 16

Dual GPU, 48GB total VRAM:

vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

Memory-constrained (reduce context to fit):

vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager  # Disables CUDA graphs, saves ~1GB VRAM

OpenAI-Compatible API

vLLM exposes the same endpoints as the OpenAI API. Any client library or tool that targets OpenAI can point at vLLM instead.

Chat Completions

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a Linux sysadmin assistant."},
      {"role": "user", "content": "How do I check disk I/O bottlenecks?"}
    ],
    "temperature": 0.7,
    "max_tokens": 512,
    "stream": true
  }'

Python Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # vLLM ignores this but the client requires it
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in 3 sentences."}],
    temperature=0.3
)
print(response.choices[0].message.content)

Supported Endpoints

• /v1/chat/completions – chat-style inference
• /v1/completions – text completion
• /v1/models – list loaded models
• /v1/embeddings – text embeddings (model-dependent)
• /health – server health check

GPU Memory Management

vLLM’s memory usage breaks down into two components: model weights and KV cache. Understanding this split is critical for capacity planning.

# Check current GPU memory before starting vLLM
nvidia-smi --query-gpu=memory.total,memory.used,memory.free \
  --format=csv,noheader,nounits

Approximate VRAM requirements (FP16):

The --gpu-memory-utilization flag controls what fraction of total VRAM vLLM claims. Setting it to 0.90 leaves 10% headroom for the CUDA runtime and other processes. If you see out-of-memory errors, reduce this value or lower --max-model-len.

Monitor memory during operation:

# Watch GPU utilization in real time
watch -n 1 nvidia-smi

# vLLM also exposes Prometheus metrics
curl http://localhost:8000/metrics | grep vllm_gpu

Benchmarks: vLLM vs Ollama Under Concurrent Load

These numbers are from a test rig with an RTX 4090 (24GB), running Llama-3.1-8B-Instruct (FP16 for vLLM, Q4_K_M for Ollama), 512-token output length, measured with a simple load generator sending requests in parallel.

At a single user, throughput is roughly equivalent. The gap widens dramatically with concurrency because vLLM’s continuous batching keeps the GPU saturated while Ollama processes requests sequentially. This is the core reason to choose vLLM for multi-user deployments.

Note: Ollama uses GGUF quantization (Q4_K_M) while vLLM uses FP16 here. The comparison is intentionally apples-to-oranges on precision because it reflects real deployment choices – you would typically run quantized models in Ollama and FP16/AWQ in vLLM.

Docker Deployment

Docker is the recommended deployment method for production use. The official image includes all CUDA dependencies.

Basic Docker Run

docker run -d \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=${HF_TOKEN} \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 4096

Docker Compose

# docker-compose.yml
services:
  vllm:
    image: vllm/vllm-openai:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    volumes:
      - huggingface-cache:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --host 0.0.0.0
      --port 8000
      --max-model-len 4096
      --gpu-memory-utilization 0.90
    restart: unless-stopped

volumes:
  huggingface-cache:

# Start
docker compose up -d

# Check logs
docker compose logs -f vllm

# Verify
curl http://localhost:8000/v1/models

Systemd Service (Non-Docker)

For bare-metal deployments, create a systemd unit:

# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM Inference Server
After=network.target

[Service]
Type=simple
User=vllm
Environment=HF_TOKEN=hf_your_token
ExecStart=/home/vllm/vllm-env/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 --max-model-len 4096
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now vllm.service
sudo systemctl status vllm.service

Practical Considerations

Model compatibility: vLLM supports most Hugging Face Transformers models, plus AWQ and GPTQ quantized variants. It does not support GGUF files – that is llama.cpp/Ollama territory. Check the vLLM documentation for the current model support matrix.

No CPU fallback: vLLM requires an NVIDIA or AMD GPU. There is no CPU-only mode. If you need CPU inference, use Ollama or llama.cpp.

Startup time: Loading a 7-8B model takes 30-60 seconds. The 70B AWQ variant takes 2-5 minutes. Plan for this in your deployment strategy – use health checks to delay traffic until the model is loaded.

Scaling: For higher throughput, use tensor parallelism across multiple GPUs rather than running multiple vLLM instances. This gives each request access to all GPUs rather than partitioning capacity.

vLLM is the right tool when you need to serve LLMs to multiple users from dedicated GPU hardware. For everything else – desktop experimentation, CPU-only machines, quick model testing – Ollama remains the simpler choice.