How to Install vLLM on Ubuntu 24.04 VPS: High-Throughput LLM Inference Server

Running a large language model in production is a very different problem than running one for personal use. The moment you need to serve dozens or hundreds of concurrent users, pass through structured outputs, or squeeze more tokens per second out of an expensive GPU, tools like Ollama and plain Hugging Face transformers stop being enough. This is where vLLM earns its reputation as the reference high-throughput inference server for open models.

This guide walks you through installing vLLM on Ubuntu 24.04 LTS from a clean VPS, configuring CUDA and NVIDIA drivers, serving a model over an OpenAI-compatible REST API, and hardening the deployment with systemd, Nginx, and TLS.

Need a GPU VPS? Our CloudCore Business plans include high-bandwidth NVMe storage and low-latency networking ideal for attaching to GPU workers. For CPU-only evaluation, the same plan will boot vLLM in CPU mode so you can prototype before moving to a GPU host.

Table of Contents

What is vLLM?

vLLM is an open-source LLM inference and serving library originally developed at UC Berkeley's Sky Computing Lab. Its defining contribution is PagedAttention, an algorithm that manages the key/value (KV) cache the same way an operating system manages virtual memory: in fixed-size pages that can be allocated, shared, and freed on demand. This one design decision eliminates the memory fragmentation that used to waste 60-80% of GPU RAM in naive batched inference, and unlocks the second feature vLLM is famous for: continuous batching, a scheduler that injects new requests into the active batch the moment existing ones free up a slot.

The practical result is throughput that tends to land somewhere between 10x and 24x higher than a vanilla Hugging Face transformers.generate loop on the same hardware, with lower p99 latency at the same concurrency.

vLLM is model-agnostic. It ships first-class support for most popular open architectures: Llama 2/3/3.1/3.2, Mistral and Mixtral, Qwen 2/2.5/3, Gemma 2/3, DeepSeek V2/V3, Yi, Phi-3/4, Falcon, Command-R, StableLM, Baichuan, MPT, Bloom, and many more. Multi-modal models like LLaVA, Phi-3-Vision, and Qwen2-VL are supported for image + text inputs. Embedding models (BGE, E5) and reranker models are served through the same API surface.

On the infrastructure side, vLLM supports NVIDIA CUDA (primary), AMD ROCm, Intel XPU, AWS Neuron (Inferentia/Trainium), Google TPU, and a CPU backend for evaluation. It scales horizontally with tensor parallelism (split a single model across multiple GPUs) and pipeline parallelism (split across multiple nodes), and the server exposes a fully OpenAI-compatible REST API so any SDK written against openai in Python, TypeScript, Go, or Rust can point at your vLLM endpoint with a one-line base URL change.

Why Self-Host LLM Inference?

The appeal of managed LLM APIs is obvious: no hardware, no scaling, no drivers. The cost curve is less obvious. The moment you cross a few million tokens per day, or have to keep prompts and completions on your own infrastructure for legal reasons, operating your own inference server starts to look dramatically cheaper and safer.

• Data privacy. Prompts, retrieved documents, and completions never leave your VPS. No third-party retention policy, no training-data contamination risk, no vendor ToS changes that suddenly prohibit a use case.
• Predictable cost. A GPU VPS is a flat monthly line item. Tokens are free at the margin. A 24 GB GPU serving a 7B model can comfortably push 2,000-6,000 output tokens per second under continuous batching, which at typical OpenAI-equivalent pricing would cost thousands per month.
• No rate limits. You are not competing with every other customer of a shared API for capacity during peak hours.
• Custom and gated models. Serve fine-tunes, LoRA adapters, domain-specific models, and multimodal checkpoints that no managed provider exposes.
• Deterministic latency. Network round-trips disappear when the app and the inference server share a VPC or a host.
• Regulatory fit. GDPR, HIPAA-style workflows, EU AI Act record-keeping, and sovereign-cloud requirements are far easier to satisfy on infrastructure you control.

Rough cost comparison

Prerequisites

• A VPS running Ubuntu 24.04 LTS with root or sudo access
• An NVIDIA GPU with compute capability 7.0 or higher (Volta V100, Turing T4/RTX 20, Ampere A10/A30/A100/RTX 30, Ada L4/L40/RTX 40, Hopper H100) and at least 16 GB VRAM for 7B models, or 48 GB+ for 70B models
• At least 32 GB system RAM and 100 GB NVMe for model weights and KV cache swapping
• Python 3.10, 3.11, or 3.12 (vLLM wheels are published for these versions)
• A Hugging Face account and access token if you plan to serve gated models like Llama 3

Recommended plan: CloudCore Business

The CloudCore Business plan gives you a fast CPU head node with the networking and storage characteristics vLLM needs alongside a GPU worker. If you do not yet have CUDA or Python 3 installed on your box, start with our companion guides:

- How to Install CUDA on Ubuntu 24.04

- How to Install Python on Ubuntu 24.04

SSH into your server:

ssh root@your-server-ip

Step 1: Update Ubuntu and Install Build Tools

Start with a clean, fully patched system and the compilers vLLM needs to build any native extensions that are not covered by a prebuilt wheel.

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential git curl wget ca-certificates \
  pkg-config libssl-dev libffi-dev \
  software-properties-common gnupg lsb-release

If the kernel was updated, reboot before continuing so the NVIDIA driver loads against the currently running kernel:

sudo reboot

Reconnect via SSH after a minute.

Step 2: Install the NVIDIA Driver and CUDA Toolkit

vLLM requires a working CUDA runtime. Prebuilt vLLM wheels on PyPI are compiled against CUDA 12.1, so matching or newer CUDA runtime drivers are required.

Install the recommended NVIDIA driver

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

After reconnecting, confirm the driver is loaded:

nvidia-smi

Expected output (abbreviated):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07    Driver Version: 550.90.07    CUDA Version: 12.4                |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          Off | 00000000:00:05.0   Off |                    0 |
| N/A   32C    P0              45W / 300W |       0MiB / 81920MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+

You need driver 550+ (which exposes CUDA 12.4) for best compatibility with vLLM.

Install the CUDA 12.1 toolkit

The driver ships with a compatible runtime, but installing the full toolkit gives you nvcc, profiling tools, and the libraries some vLLM optional features (such as FlashInfer) want at build time.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-1

Add CUDA to your shell profile:

echo 'export PATH=/usr/local/cuda-12.1/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Verify:

nvcc --version

Expected output:

nvcc: NVIDIA (R) Cuda compiler driver
Cuda compilation tools, release 12.1, V12.1.105

If you need a step-by-step CUDA walkthrough, see How to Install CUDA on Ubuntu 24.04.

CPU-only readers: you can skip this step entirely and jump to CPU-Only Mode after completing Steps 3-5. Expect ~1-10 tokens/sec on a modern server CPU; acceptable for evaluation, not for production.

Step 3: Create a Python Virtual Environment

Never install vLLM into the system Python. The dependency footprint is heavy (PyTorch, NCCL, xformers, FlashAttention) and upgrading the OS can silently break it. Use a dedicated virtualenv.

sudo apt install -y python3.12 python3.12-venv python3-pip

Create a dedicated service user (optional but recommended) and a venv under /opt:

sudo useradd --system --create-home --shell /bin/bash vllm
sudo mkdir -p /opt/vllm
sudo chown vllm:vllm /opt/vllm
sudo -u vllm python3.12 -m venv /opt/vllm/venv

Activate the venv (for the interactive setup steps only — the systemd unit will call the binary directly later):

sudo -u vllm -i
source /opt/vllm/venv/bin/activate
pip install --upgrade pip setuptools wheel

Step 4: Install vLLM via pip

With the venv active, install vLLM. The prebuilt wheel pulls a pinned PyTorch build compiled against CUDA 12.1.

pip install vllm

The install takes 5-15 minutes and downloads roughly 6-8 GB. Among the packages it pulls you will see torch, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-nccl-cu12, xformers, transformers, tokenizers, and fastapi.

Verify the install:

python -c "import vllm; print(vllm.__version__)"

Expected output:

0.6.4

And confirm CUDA is visible to PyTorch:

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count(), torch.version.cuda)"

Expected output:

True 1 12.1

If torch.cuda.is_available() returns False, your driver or CUDA install is not matched — rerun nvidia-smi and confirm the driver version is 550+.

Step 5: Authenticate with Hugging Face

Many production-quality open models — Llama 3.x, Gemma, Mistral Instruct — are gated on Hugging Face. You must accept their license on the Hugging Face website and authenticate your server before vLLM can download them.

mkdir -p /opt/vllm/.cache/huggingface
echo 'hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' > /opt/vllm/.cache/huggingface/token
chmod 600 /opt/vllm/.cache/huggingface/token

Or export it as an environment variable (the systemd unit below does this via a drop-in environment file):

echo 'export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"' >> ~/.bashrc
source ~/.bashrc

You can also set HF_HOME=/opt/vllm/.cache/huggingface if you want model weights stored on a larger mounted disk.

Step 6: Launch the OpenAI-Compatible API Server

vLLM ships an OpenAI-compatible HTTP server as vllm.entrypoints.openai.api_server. A single command boots it on port 8000.

Still inside the vllm user shell with the venv active:

vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Equivalently, the long-form module invocation:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Flag highlights:

• --model — Hugging Face repo ID or local path. vLLM downloads and caches on first run.
• --dtype auto — use the native dtype of the checkpoint (bfloat16 on Ampere/Hopper, float16 on Turing).
• --max-model-len — hard cap on sequence length (prompt + completion). Controls KV cache size per request.
• --gpu-memory-utilization 0.90 — fraction of VRAM vLLM is allowed to preallocate for KV cache paging. 0.85-0.92 is typical.
• --host 0.0.0.0 — listen on all interfaces (bind to 127.0.0.1 if you will only proxy through Nginx on the same host).

INFO ... Application startup complete.
INFO ... Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Leave this terminal running while you test in the next step.

Step 7: Test the API

The server speaks the OpenAI Chat Completions and Completions protocols. From a second SSH session:

Health check

curl http://localhost:8000/health

Expected output:

{"status":"ok"}

List loaded models

curl http://localhost:8000/v1/models

Expected output (abbreviated):

{"object":"list","data":[{"id":"meta-llama/Meta-Llama-3.1-8B-Instruct","object":"model",...}]}

Chat completion

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a concise senior SRE."},
      {"role": "user", "content": "Summarize PagedAttention in two sentences."}
    ],
    "max_tokens": 150,
    "temperature": 0.3
  }'

Expected output:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "PagedAttention manages the attention key/value cache in fixed-size pages, mirroring OS virtual memory, which eliminates fragmentation and allows the KV cache to be shared across requests. This lets vLLM pack far more concurrent sequences into the same GPU memory than a contiguous KV cache would allow."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 36, "completion_tokens": 61, "total_tokens": 97}
}

From the Python OpenAI SDK

from openai import OpenAI
client = OpenAI(base_url="http://your-server-ip:8000/v1", api_key="dummy")
resp = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Every SDK that speaks OpenAI works — LangChain, LlamaIndex, Instructor, Vercel AI SDK, LiteLLM, Continue.dev — simply by swapping the base URL.

Step 8: Tensor Parallelism and PagedAttention

PagedAttention is always on — there is no flag to enable it. What you can tune is tensor parallelism, which shards a single model across multiple GPUs so models that would never fit on one card can be served on a multi-GPU host.

To serve Llama 3.1 70B on a 2x A100 80GB node:

vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

Key points:

• --tensor-parallel-size must evenly divide the number of attention heads. Llama 3.1 70B has 64 heads, so 1, 2, 4, 8 are all valid.
• All GPUs should be on the same node and connected via NVLink or fast PCIe — inter-GPU communication during every forward pass is the main cost.
• For multi-node serving add --pipeline-parallel-size N, though tensor parallel inside a node plus pipeline across nodes is the standard recipe.

watch -n 1 nvidia-smi

You should see VRAM utilization climb to roughly --gpu-memory-utilization (90% by default) on every shard and SM utilization spike into the 80-99% range when requests are in-flight.

Step 9: Quantization (AWQ and GPTQ)

Quantization reduces model size and increases throughput at the cost of a small quality loss. vLLM supports several formats. The two you will encounter most often are AWQ (Activation-aware Weight Quantization) and GPTQ (Generative Pre-trained Transformer Quantization). Both reduce weights from 16-bit to 4-bit, cutting VRAM by roughly 3.5x.

Serve an AWQ model

vllm serve TheBloke/Llama-2-13B-chat-AWQ \
  --quantization awq \
  --dtype float16 \
  --max-model-len 4096

Serve a GPTQ model

vllm serve TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
  --quantization gptq \
  --dtype float16 \
  --max-model-len 8192

Other supported quantizations

• --quantization fp8 — Hopper-native 8-bit floating point (H100, H200, L40S)
• --quantization marlin — fast 4-bit kernels for Ampere+
• --quantization squeezellm — dense and sparse mixed quantization
• --quantization bitsandbytes — 8-bit and 4-bit (NF4) via bitsandbytes

Rough VRAM footprint (7B model)

Step 10: Continuous Batching Tuning

Continuous batching is enabled by default. The two knobs that matter are the max number of sequences and the max number of batched tokens.

vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-chunked-prefill

• --max-num-seqs — the maximum number of requests the scheduler will keep in the running batch. Higher = more concurrency, more VRAM pressure.
• --max-num-batched-tokens — token budget per scheduler step. A good starting point is equal to --max-model-len.
• --enable-chunked-prefill — breaks long prompts into chunks that interleave with decoding, keeping decode latency low when a large prompt arrives mid-stream.

Benchmark your config

vLLM ships a benchmark script:

python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-8B-Instruct &
in another shell:
python /opt/vllm/venv/lib/python3.12/site-packages/vllm/benchmarks/benchmark_serving.py \
  --backend openai \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --num-prompts 500 \
  --request-rate 10

Sample result on a single A100 80GB running Llama 3.1 8B bfloat16:

Throughput: 3,240 output tokens/s
Mean TTFT: 180 ms
P99 TTFT: 420 ms

Compare runs as you change --max-num-seqs and --gpu-memory-utilization to find your throughput ceiling.

Step 11: Create a systemd Service

To keep vLLM running across reboots and crashes, wrap it in a systemd unit.

Create an environment file (keeps the HF token out of the unit file):

sudo tee /etc/vllm.env > /dev/null <<'EOF'
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
HF_HOME=/opt/vllm/.cache/huggingface
VLLM_WORKER_MULTIPROC_METHOD=spawn
EOF
sudo chown root:vllm /etc/vllm.env
sudo chmod 640 /etc/vllm.env

Create the unit:

sudo tee /etc/systemd/system/vllm.service > /dev/null <<'EOF'
[Unit]
Description=vLLM OpenAI-compatible Inference Server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=vllm
Group=vllm
EnvironmentFile=/etc/vllm.env
WorkingDirectory=/opt/vllm
ExecStart=/opt/vllm/venv/bin/python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --enable-chunked-prefill
Restart=on-failure
RestartSec=10
Generation can take a while; give it room to shut down cleanly
TimeoutStopSec=60
vLLM likes lots of file descriptors
LimitNOFILE=1048576
[Install]
WantedBy=multi-user.target
EOF

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
sudo systemctl status vllm

Tail the logs:

sudo journalctl -u vllm -f

On first boot it will spend several minutes downloading the model; subsequent starts take 30-90 seconds.

Step 12: Nginx Reverse Proxy with TLS

Expose the API to the public internet through Nginx with a free Let's Encrypt certificate and optional bearer-token authentication.

Install Nginx and Certbot

sudo apt install -y nginx certbot python3-certbot-nginx

Point a DNS A record for llm.yourdomain.com at the server before continuing.

Nginx configuration

sudo tee /etc/nginx/sites-available/vllm > /dev/null <<'EOF'
Rate-limit to protect the GPU
limit_req_zone $binary_remote_addr zone=vllm_rl:10m rate=30r/s;
map $http_authorization $vllm_auth_ok {
    default 0;
    "Bearer sk-your-long-random-shared-secret" 1;
}
server {
    listen 80;
    server_name llm.yourdomain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name llm.yourdomain.com;
    ssl_certificate     /etc/letsencrypt/live/llm.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.yourdomain.com/privkey.pem;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-Frame-Options DENY always;
    client_max_body_size 16m;
    location = /health {
        proxy_pass http://127.0.0.1:8000/health;
    }
    location / {
        if ($vllm_auth_ok = 0) { return 401; }
        limit_req zone=vllm_rl burst=60 nodelay;
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Streaming / SSE
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
        proxy_set_header Connection "";
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/vllm /etc/nginx/sites-enabled/
sudo nginx -t

Issue a certificate

sudo certbot --nginx -d llm.yourdomain.com --non-interactive --agree-tos -m [email protected]
sudo systemctl reload nginx

Test from the outside

curl https://llm.yourdomain.com/v1/chat/completions \
  -H "Authorization: Bearer sk-your-long-random-shared-secret" \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Meta-Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"ping"}]}'

You now have a production-shaped deployment: vLLM under systemd, bound to localhost, fronted by Nginx with TLS, HTTP/2, streaming, a shared-secret bearer token, and rate limiting.

For a stronger auth model, replace the shared token with a dedicated gateway such as LiteLLM or an Nginx + JWT setup that issues per-user keys with spend limits.

CPU-Only Mode

If you do not have a GPU yet and want to evaluate vLLM's API and scheduler on a CPU VPS, use the CPU backend.

Install the CPU build from source (the PyPI wheel is CUDA-only):

sudo apt install -y build-essential cmake
pip uninstall -y vllm
VLLM_TARGET_DEVICE=cpu pip install -v git+https://github.com/vllm-project/vllm.git

Run with a small model:

vllm serve Qwen/Qwen2.5-0.5B-Instruct --device cpu --dtype float32 --max-model-len 2048

Expect single-digit tokens per second. This mode is useful for smoke-testing integrations but is not suitable for production traffic. When you are ready to move to real hardware, spin up a GPU host or pair your CloudCore Business head node with a GPU worker and switch the --device and --dtype flags back to GPU defaults.

For a lighter-weight alternative that works well on CPU, consider Ollama — different design goals, better CPU story.

Troubleshooting

sudo journalctl -u vllm -f

Run a one-off debug server in the foreground with verbose logging:

sudo systemctl stop vllm
sudo -u vllm -i
source /opt/vllm/venv/bin/activate
VLLM_LOGGING_LEVEL=DEBUG vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --enforce-eager

FAQ

What is vLLM and how does it differ from Ollama?

vLLM is a production-oriented LLM inference server built around PagedAttention and continuous batching. It is optimized for serving many concurrent users from one or more GPUs with maximum throughput and exposes an OpenAI-compatible REST API. Ollama is a developer-experience-first tool for running a model locally on CPU or a single GPU with an easy CLI. If you are building a product that will serve real traffic and you have GPU capacity, use vLLM. If you are prototyping on a laptop or a single-user VPS, Ollama is faster to set up.

Do I really need a GPU?

For anything beyond evaluation, yes. vLLM's architectural advantages — PagedAttention, continuous batching, tensor parallelism — are designed around GPU memory and compute. On CPU you get a working OpenAI API but only a few tokens per second. A single NVIDIA A10 24GB or L4 24GB is enough to serve a quantized 7B model at hundreds of tokens per second; an A100 80GB or H100 can push multi-thousand tokens per second per GPU.

Which quantization should I pick: AWQ or GPTQ?

Both are 4-bit weight-only schemes. In practice AWQ usually edges out GPTQ on instruction-following benchmarks and has better community tooling right now. If a model is published in both formats, start with AWQ. If only GPTQ is available (common for older checkpoints), GPTQ is perfectly serviceable. For Hopper-generation GPUs (H100, H200, L40S), FP8 often beats both 4-bit formats because the hardware has native FP8 Tensor Cores.

How does continuous batching actually help?

A traditional batched inference server waits until it has N requests queued, pads them to the length of the longest prompt, runs one forward pass, then repeats. Short requests block long ones and padding wastes GPU cycles. Continuous batching treats every scheduler tick independently: once one sequence finishes a token, a new request can fill its slot. Combined with PagedAttention (which stores each sequence's KV cache in shareable pages rather than a contiguous slab), the net effect is that the GPU stays close to 100% utilization across a mixed workload. Real-world throughput gains over naive batched transformers are typically 10-24x.

How do I scale beyond one GPU?

Inside a single host, use --tensor-parallel-size N to shard a single model across N GPUs. This is how you fit a 70B model onto 2x A100 80GB or 4x RTX 4090. Across multiple hosts, combine tensor parallelism within each node with --pipeline-parallel-size across nodes, or run multiple independent vLLM replicas behind a load balancer (LiteLLM or simple Nginx round-robin). For most workloads, "scale up with TP before scaling out" is the right order.

Next Steps

You now have a production-shaped vLLM deployment. From here, a few directions worth exploring:

• Pair vLLM with a gateway. Drop LiteLLM in front of vLLM to add per-user API keys, spend budgets, routing between multiple backend models, and unified logging — all while preserving the OpenAI API surface.
• Add embeddings and rerankers. Run a second vLLM process on another port serving an embedding model like BAAI/bge-large-en-v1.5 and a reranker like BAAI/bge-reranker-v2-m3 to power a full RAG stack on the same box.
• Compare with Ollama for CPU/edge use cases. See How to Install Ollama on Ubuntu 24.04 when you need single-user, CPU-friendly inference.
• Harden the Python and CUDA stack. If you have not already, follow How to Install CUDA on Ubuntu 24.04 and How to Install Python on Ubuntu 24.04 to keep your base image reproducible.
• Read the upstream docs. The official vLLM documentation is excellent and covers advanced topics like speculative decoding, LoRA adapter hot-swapping, structured outputs with guided_json, prefix caching, and multi-LoRA serving.

Ready to run vLLM in production?

Start with our CloudCore Business VPS as your control-plane host — NVMe storage, low-latency networking, and snapshot backups — and attach a GPU worker when you are ready to serve real traffic. No per-token fees. No rate limits. Your data never leaves your infrastructure.