How to Build a Complete Self-Hosted AI Stack on Ubuntu 24.04 — Ollama + Open WebUI + RAG + Vector DB

Running a production-grade AI stack used to mean signing multi-year contracts with hyperscalers, paying per token for every query, and sending your data to third parties. That world is over. With a modest VPS and a single evening of work you can now stand up a private ChatGPT-equivalent platform — complete with a polished web UI, retrieval-augmented generation over your own documents, a vector database, an OpenAI-compatible API gateway, and HTTPS reverse proxy — that costs a few dollars a month to run and never leaks a single token to a third party.

This guide walks you through the full build on Ubuntu 24.04: architecture, cost comparison, every component, a complete docker-compose.yml, nginx subdomain routing with SSL, backups, monitoring, and a full list of optional upgrades (LibreChat, Dify, SearXNG). By the end you will have a working stack at chat.yourdomain.com, api.yourdomain.com, and vector.yourdomain.com serving real inference and real RAG answers.

Want to skip provisioning? Our GPU Server plans come preloaded with NVIDIA drivers, CUDA, and Docker — perfect for this stack. Launch a GPU VPS now and be prompting in under 10 minutes.

Table of Contents

Architecture Overview

The stack is designed as six cooperating containers behind an nginx reverse proxy. Each container owns one concern, communicates over an internal Docker network, and persists state to a named volume on the host.

Internet
                           |
                           | HTTPS (443)
                           v
                 +-------------------+
                 |      nginx        |   <- Let's Encrypt SSL
                 |  (host network)   |
                 +-------------------+
                    |       |      |
         chat.     | api. | vector.*
                    v       v      v
   +------------------+  +---------+  +-----------+
   |   Open WebUI     |  | LiteLLM |  |  Qdrant   |
   |  (port 8080)     |  | (4000)  |  | (6333/4)  |
   +------------------+  +---------+  +-----------+
            |                 |             ^
            |                 |             |
            v                 v             |
     +-------------+    +-----------+       |
     |   Ollama    |<---|  models   |       |
     | (11434)     |    +-----------+       |
     +-------------+                        |
            ^                                |
            |  embeddings / chunks           |
            |                                |
     +-------------+                        |
     |   Docling   |------------------------+
     | (document   |  parsed text
     |  processor) |
     +-------------+
            ^
            |  PDFs, DOCX, HTML
            |
     +-------------+
     |  Uploads    |
     +-------------+

Data flow for a RAG query:

Data flow for document ingestion:

Everything lives on one VPS. Nothing leaves your network unless you explicitly connect LiteLLM to an external API.

Why Build a Full Self-Hosted AI Stack?

Data sovereignty. Every token of every prompt and every document you upload stays on hardware you control. For regulated industries (healthcare, legal, finance) and for companies working with proprietary information this is non-negotiable. There is no "the vendor changed their terms" risk, no "your prompts may be used for training" fine print, no "breach at a subprocessor" disclosure letter.

Predictable flat-rate pricing. One monthly VPS bill replaces a metered-token invoice that scales linearly with usage. Once the box is paid for, inference is effectively free. You can run a team of thirty people hammering the stack all day and the bill does not move.

No rate limits. You can batch-process ten thousand documents overnight without getting throttled or cut off. You can fine-tune a retrieval pipeline with a million test queries and not worry about exhausting a quota.

Model flexibility. You are not tied to whatever models a single vendor exposes. You can pull Llama 3.1, Gemma, Qwen, Mistral, DeepSeek, or any GGUF from Hugging Face and swap between them with a single CLI command. You can run a small model for fast drafts and a 70B model for hard questions, side by side.

Integration freedom. Because every component exposes an OpenAI-compatible API, any tool that speaks the OpenAI format — n8n, LangChain, LlamaIndex, Cursor, Continue.dev, custom app code — plugs in without modification. You get the entire OpenAI ecosystem pointing at your own hardware.

Learning and control. Running the stack teaches you the mechanics of modern AI systems — tokenization, embeddings, chunking, retrieval, prompt templating, generation sampling — in a way that clicking "Send" on api.openai.com never will.

Cost Comparison: Self-Hosted vs OpenAI + Pinecone

Assume a modest team use case: 2 million input tokens and 500,000 output tokens per day, plus RAG over a 100,000-chunk knowledge base.

OpenAI + Pinecone (cloud)

The break-even point against OpenAI Plus ($20/user/mo) is roughly two active users. Against pay-per-token API usage at any serious scale, the break-even is measured in days.

Where cloud still wins: peak coding performance from frontier models (GPT-4o, Claude Opus) remains ahead of any open-weights model you can run on a single VPS. The recommended pattern is hybrid: run the stack self-hosted for 80% of queries and route the remaining 20% of truly hard questions to a cloud API through LiteLLM. You still cut the bill by 70% and keep the high-end escape hatch.

Prerequisites and VPS Sizing

Minimum (small team, 7B–9B models only):

• 6 vCPU
• 16 GB RAM
• 120 GB NVMe SSD
• Ubuntu 24.04 LTS
• Public IPv4
• Domain with DNS access

• 8 vCPU
• 24 GB RAM
• 240 GB NVMe SSD
• Ubuntu 24.04 LTS

• 8 vCPU
• 32 GB RAM
• NVIDIA GPU with 24 GB+ VRAM (RTX 4090, A5000, L40)
• 500 GB NVMe SSD

You will also need: a domain name (any registrar), basic SSH / Linux familiarity, and a few hours.

The Components Explained

Ollama — the LLM runtime. Ollama wraps llama.cpp with a clean CLI and REST API. It handles model downloads, quantization, GPU offload, and concurrent request scheduling. One command pulls a model, one command serves it. We covered Ollama in depth in How to Install Ollama on Ubuntu 24.04.

Open WebUI — the ChatGPT-style frontend. A polished, multi-user web interface that looks and feels like ChatGPT. Out of the box it supports chat histories, model switching, image uploads, function calling, multi-user accounts with RBAC, and — crucially — a built-in RAG pipeline with knowledge-base management. It talks to Ollama natively and to any OpenAI-compatible endpoint.

Qdrant — the vector database. A Rust-native vector DB with excellent performance-per-dollar, a clean REST + gRPC API, payload filtering, and a web dashboard. Open WebUI ships with an embedded Chroma instance but we swap in Qdrant because it scales to hundreds of millions of vectors, survives restarts cleanly, and can be shared by other apps on the stack.

Docling — the document processor. IBM's open-source document parser that turns PDFs, DOCX, PPTX, HTML, and images into clean, structured Markdown with tables, code blocks, and layout preserved. Much better than pypdf or Tika for real-world documents. Open WebUI can be configured to route uploads through Docling before chunking.

LiteLLM — the OpenAI-compatible API gateway. A thin proxy that exposes every model behind a single OpenAI-format endpoint (/v1/chat/completions, /v1/embeddings). Point it at Ollama, Anthropic, OpenAI, Azure, Bedrock, and Gemini simultaneously and you get one API key, one URL, and per-model rate limits, budgets, and logging. Essential if other applications (n8n, Cursor, your SaaS backend) need to call the stack.

nginx — the TLS reverse proxy. Terminates HTTPS, routes each subdomain to the right container, handles WebSocket upgrades for streaming responses, and enforces rate limits.

Step 1: Prepare the Server

SSH in as a non-root sudo user and update the system.

sudo apt update && sudo apt upgrade -y
sudo apt install -y ca-certificates curl gnupg lsb-release ufw fail2ban git jq

Install Docker Engine and the Compose plugin from Docker's official APT repo (not the older docker.io package):

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Configure the firewall:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

If you are on a GPU host, install the NVIDIA Container Toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

The final nvidia-smi call should print your GPU's full name and memory. If it does, GPU passthrough works.

Point three DNS A records at the server's public IP:

chat.yourdomain.com     A   203.0.113.42
api.yourdomain.com      A   203.0.113.42
vector.yourdomain.com   A   203.0.113.42

Step 2: Directory Layout and Environment File

Create a clean working directory:

sudo mkdir -p /opt/ai-stack
sudo chown $USER:$USER /opt/ai-stack
cd /opt/ai-stack
mkdir -p configs/litellm data/uploads backups scripts

Create /opt/ai-stack/.env:

# --- general ---
TZ=UTC
DOMAIN=yourdomain.com
--- Open WebUI ---
WEBUI_SECRET_KEY=REPLACE_WITH_openssl_rand_hex_32
WEBUI_AUTH=true
ENABLE_SIGNUP=false
DEFAULT_USER_ROLE=pending
--- Ollama ---
OLLAMA_HOST=0.0.0.0
OLLAMA_NUM_PARALLEL=2
OLLAMA_MAX_LOADED_MODELS=2
OLLAMA_KEEP_ALIVE=30m
--- Qdrant ---
QDRANT_API_KEY=REPLACE_WITH_openssl_rand_hex_32
--- LiteLLM ---
LITELLM_MASTER_KEY=sk-REPLACE_WITH_openssl_rand_hex_32
LITELLM_SALT_KEY=REPLACE_WITH_openssl_rand_hex_32
--- External APIs (optional) ---
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GEMINI_API_KEY=

Generate the three secrets:

for k in WEBUI_SECRET_KEY QDRANT_API_KEY LITELLM_MASTER_KEY LITELLM_SALT_KEY; do
  echo "$k=$(openssl rand -hex 32)"
done

Paste those values into .env and chmod it:

chmod 600 /opt/ai-stack/.env

Create the LiteLLM config at /opt/ai-stack/configs/litellm/config.yaml:

model_list:
  - model_name: llama3.1
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://ollama:11434
  - model_name: gemma2
    litellm_params:
      model: ollama/gemma2:9b
      api_base: http://ollama:11434
  - model_name: nomic-embed
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://ollama:11434
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings:
  drop_params: true
  request_timeout: 600
  telemetry: false

Step 3: The Complete docker-compose.yml

Save this as /opt/ai-stack/docker-compose.yml:

name: ai-stack
networks:
  ai-net:
    driver: bridge
volumes:
  ollama-data:
  openwebui-data:
  qdrant-data:
  litellm-data:
services:
  # -----------------------------
  # Ollama — LLM inference server
  # -----------------------------
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    environment:
      - OLLAMA_HOST=${OLLAMA_HOST}
      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL}
      - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS}
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE}
    volumes:
      - ollama-data:/root/.ollama
    networks:
      - ai-net
    # Uncomment for GPU hosts:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 30s
      timeout: 10s
      retries: 3
  # -----------------------------
  # Qdrant — vector database
  # -----------------------------
  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    restart: unless-stopped
    environment:
      - QDRANT__SERVICE__API_KEY=${QDRANT_API_KEY}
      - QDRANT__SERVICE__ENABLE_CORS=true
      - QDRANT__LOG_LEVEL=INFO
      - QDRANT__STORAGE__OPTIMIZERS__DEFAULT_SEGMENT_NUMBER=2
    volumes:
      - qdrant-data:/qdrant/storage
    networks:
      - ai-net
    healthcheck:
      test: ["CMD-SHELL", "bash -c ':> /dev/tcp/127.0.0.1/6333' || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
  # -----------------------------
  # Docling — document processor
  # -----------------------------
  docling:
    image: quay.io/docling-project/docling-serve:latest
    container_name: docling
    restart: unless-stopped
    environment:
      - DOCLING_SERVE_ENABLE_UI=false
      - DOCLING_SERVE_MAX_SYNC_WAIT=120
    networks:
      - ai-net
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:5001/health"]
      interval: 30s
      timeout: 10s
      retries: 3
  # -----------------------------
  # Open WebUI — chat frontend
  # -----------------------------
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      ollama:
        condition: service_healthy
      qdrant:
        condition: service_healthy
    environment:
      - TZ=${TZ}
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - WEBUI_AUTH=${WEBUI_AUTH}
      - ENABLE_SIGNUP=${ENABLE_SIGNUP}
      - DEFAULT_USER_ROLE=${DEFAULT_USER_ROLE}
      - OLLAMA_BASE_URL=http://ollama:11434
      - OPENAI_API_BASE_URL=http://litellm:4000/v1
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      # RAG / Vector DB
      - VECTOR_DB=qdrant
      - QDRANT_URI=http://qdrant:6333
      - QDRANT_API_KEY=${QDRANT_API_KEY}
      - RAG_EMBEDDING_ENGINE=ollama
      - RAG_EMBEDDING_MODEL=nomic-embed-text
      - RAG_OLLAMA_BASE_URL=http://ollama:11434
      - CHUNK_SIZE=1200
      - CHUNK_OVERLAP=150
      # Docling
      - CONTENT_EXTRACTION_ENGINE=docling
      - DOCLING_SERVER_URL=http://docling:5001
    volumes:
      - openwebui-data:/app/backend/data
      - ./data/uploads:/app/backend/data/uploads
    ports:
      - "127.0.0.1:8080:8080"
    networks:
      - ai-net
  # -----------------------------
  # LiteLLM — OpenAI-compatible gateway
  # -----------------------------
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: litellm
    restart: unless-stopped
    depends_on:
      ollama:
        condition: service_healthy
    command:
      - "--config=/app/config.yaml"
      - "--port=4000"
      - "--num_workers=2"
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - GEMINI_API_KEY=${GEMINI_API_KEY}
    volumes:
      - ./configs/litellm/config.yaml:/app/config.yaml:ro
      - litellm-data:/app/data
    ports:
      - "127.0.0.1:4000:4000"
    networks:
      - ai-net

Key design decisions:

• All public-facing ports are bound to 127.0.0.1. Nothing is reachable from the internet except via nginx, which we configure in Step 10.
• Qdrant and Ollama never publish ports — they are only reachable inside the Docker network.
• Named volumes for all state; ./data/uploads bind-mount so you can scp files in.
• Health checks gate dependency ordering so Open WebUI does not start hammering Ollama before it is ready.
• GPU support is one comment-unquote away if you have NVIDIA.

Step 4: Launch the Stack

cd /opt/ai-stack
docker compose pull
docker compose up -d
docker compose ps

All six services should show running and healthy within 30–60 seconds. Tail the logs the first time:

docker compose logs -f --tail=50

Quick local checks:

curl -s http://127.0.0.1:8080/health           # Open WebUI
curl -s http://127.0.0.1:4000/health/liveness  # LiteLLM
curl -sH "api-key: $QDRANT_API_KEY" \
  $(docker compose exec qdrant curl -s http://localhost:6333/readyz)

Step 5: Pull Your Models

Pull a chat model, an alternative chat model, and an embedding model:

# Llama 3.1 8B — balanced general chat
docker compose exec ollama ollama pull llama3.1:8b
Gemma 2 9B — Google's efficient alternative (swap in gemma3 when available on your host)
docker compose exec ollama ollama pull gemma2:9b
nomic-embed-text — the embedding model for RAG
docker compose exec ollama ollama pull nomic-embed-text
Verify
docker compose exec ollama ollama list

Optional extras worth pulling:

docker compose exec ollama ollama pull qwen2.5-coder:7b   # code generation
docker compose exec ollama ollama pull llava:7b           # multimodal (vision)
docker compose exec ollama ollama pull mistral:7b         # alternative reasoning

On a GPU host with 24 GB+ VRAM you can step up to llama3.1:70b (quantized) for production-quality answers.

Step 6: First-Run Open WebUI Setup

At this point Open WebUI is live at http://127.0.0.1:8080 but not yet reachable from outside. Before we do the nginx step, tunnel in from your laptop:

ssh -L 8080:127.0.0.1:8080 youruser@yourserver

Then open http://localhost:8080 in your browser.

Create a workspace, pick llama3.1:8b from the model dropdown, and send a test prompt. You should see the first token in under two seconds on CPU-only hardware, under 500 ms with a GPU.

Step 7: Connect External APIs via LiteLLM

LiteLLM is already running and already the default OpenAI-compatible backend for Open WebUI (see OPENAI_API_BASE_URL=http://litellm:4000/v1 in the compose file). This means every model you list in configs/litellm/config.yaml shows up as a selectable model inside Open WebUI.

To enable a cloud model, drop the API key into .env and restart:

# edit /opt/ai-stack/.env
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
docker compose up -d litellm

Verify from the host:

curl -sS http://127.0.0.1:4000/v1/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq .

You should see llama3.1, gemma2, nomic-embed, plus any cloud models you enabled.

LiteLLM also gives you per-key budgets and logging — useful once multiple users or apps are hitting the gateway. See the admin UI at http://127.0.0.1:4000 for usage dashboards.

Step 8: Configure Qdrant Collections

Open WebUI will create its own Qdrant collection automatically the first time you upload a document, so you do not need to touch Qdrant for the basic flow. But if you want a second, independent collection for a custom app, create it via the REST API:

curl -X PUT http://127.0.0.1:6333/collections/custom-kb \
  -H "api-key: $QDRANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": {
      "size": 768,
      "distance": "Cosine"
    },
    "optimizers_config": {
      "default_segment_number": 2
    },
    "hnsw_config": {
      "m": 16,
      "ef_construct": 100
    }
  }'

768 matches the dimensionality of nomic-embed-text. Use 1024 for mxbai-embed-large, 1536 for OpenAI text-embedding-3-small.

View the Qdrant dashboard at http://127.0.0.1:6333/dashboard (tunnel it like the WebUI in Step 6). You can inspect collections, browse points, and run visual similarity searches.

Step 9: Build a Working RAG Pipeline

In Open WebUI:

You are a helpful assistant. Answer using only the provided context.
If the context does not contain the answer, say "I could not find
that in the provided documents." Cite sources inline as [1], [2], etc.

Testing the pipeline end-to-end:

# Hit Open WebUI's OpenAI-compatible endpoint directly
curl http://127.0.0.1:8080/api/chat/completions \
  -H "Authorization: Bearer <your-open-webui-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role":"user","content":"What is our refund policy?"}],
    "files": [{"type":"collection","id":"company-handbook"}]
  }'

Step 10: nginx Reverse Proxy with SSL

Install nginx and certbot on the host (not as a container — we want host networking and Let's Encrypt integration):

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/conf.d/ai-stack.conf:

# --- Shared upgrade map for WebSockets ---
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
-----------------------------
chat.yourdomain.com  ->  Open WebUI
-----------------------------
server {
    listen 80;
    server_name chat.yourdomain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name chat.yourdomain.com;
    # SSL certs provisioned by certbot (Step below)
    ssl_certificate     /etc/letsencrypt/live/chat.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/chat.yourdomain.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    client_max_body_size 200M;   # allow large PDF uploads
    location / {
        proxy_pass         http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        # WebSocket / streaming
        proxy_set_header   Upgrade    $http_upgrade;
        proxy_set_header   Connection $connection_upgrade;
        proxy_buffering      off;
        proxy_read_timeout   600s;
        proxy_send_timeout   600s;
    }
}
-----------------------------
api.yourdomain.com  ->  LiteLLM
-----------------------------
server {
    listen 80;
    server_name api.yourdomain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name api.yourdomain.com;
    ssl_certificate     /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.yourdomain.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    client_max_body_size 20M;
    # Light rate limit: 60 r/s per IP with burst of 120
    limit_req_zone $binary_remote_addr zone=litellm:10m rate=60r/s;
    location / {
        limit_req zone=litellm burst=120 nodelay;
        proxy_pass         http://127.0.0.1:4000;
        proxy_http_version 1.1;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_buffering    off;
        proxy_read_timeout 600s;
    }
}
-----------------------------
vector.yourdomain.com  ->  Qdrant (optional, admin-only)
-----------------------------
server {
    listen 80;
    server_name vector.yourdomain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name vector.yourdomain.com;
    ssl_certificate     /etc/letsencrypt/live/vector.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/vector.yourdomain.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    # Restrict to your office/home IPs
    allow 203.0.113.0/24;
    deny  all;
    location / {
        # Qdrant's REST port — expose only if you know what you are doing
        proxy_pass         http://127.0.0.1:6333;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

Because Qdrant is referenced by subdomain we also need to publish its port on localhost. Add to the Qdrant service in docker-compose.yml:

ports:
      - "127.0.0.1:6333:6333"

And docker compose up -d qdrant.

Provision SSL:

sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx \
  -d chat.yourdomain.com \
  -d api.yourdomain.com \
  -d vector.yourdomain.com \
  --agree-tos -m [email protected] --redirect

Certbot auto-renews via a systemd timer; verify with sudo systemctl list-timers | grep certbot.

Visit https://chat.yourdomain.com — the Open WebUI login should load over HTTPS with a green padlock.

Step 11: Backups

Four things need backing up: Ollama models (regenerable but slow), Open WebUI database (chats, users, knowledge-base metadata), Qdrant collections (the vectors), and LiteLLM data (keys and logs).

Save this as /opt/ai-stack/scripts/backup.sh:

#!/usr/bin/env bash
set -euo pipefail
BACKUP_ROOT=/opt/ai-stack/backups
STAMP=$(date +%Y%m%d-%H%M%S)
DEST=$BACKUP_ROOT/$STAMP
mkdir -p "$DEST"
cd /opt/ai-stack
1. Open WebUI (SQLite + uploads) — stop briefly for a clean copy
echo "[*] Backing up Open WebUI..."
docker compose stop open-webui
docker run --rm \
  -v ai-stack_openwebui-data:/data:ro \
  -v "$DEST":/backup \
  alpine tar czf /backup/openwebui-data.tar.gz -C /data .
tar czf "$DEST/openwebui-uploads.tar.gz" -C data uploads
docker compose start open-webui
2. Qdrant — use its snapshot API (no downtime)
echo "[*] Snapshotting Qdrant collections..."
COLLECTIONS=$(curl -sS -H "api-key: $QDRANT_API_KEY" \
  http://127.0.0.1:6333/collections | jq -r '.result.collections[].name')
for C in $COLLECTIONS; do
  curl -sS -X POST -H "api-key: $QDRANT_API_KEY" \
    "http://127.0.0.1:6333/collections/$C/snapshots" > /dev/null
done
docker run --rm \
  -v ai-stack_qdrant-data:/data:ro \
  -v "$DEST":/backup \
  alpine tar czf /backup/qdrant-snapshots.tar.gz -C /data/snapshots .
3. LiteLLM
docker run --rm \
  -v ai-stack_litellm-data:/data:ro \
  -v "$DEST":/backup \
  alpine tar czf /backup/litellm-data.tar.gz -C /data .
4. Config files
tar czf "$DEST/configs.tar.gz" .env configs docker-compose.yml
5. Rotate — keep last 14 daily backups
find "$BACKUP_ROOT" -maxdepth 1 -type d -mtime +14 -exec rm -rf {} \;
echo "[*] Backup complete: $DEST"
du -sh "$DEST"

Note: we deliberately do not back up Ollama model files — they are large, unchanging, and trivially re-pullable. Instead, save the list:

docker compose exec -T ollama ollama list | tail -n +2 | awk '{print $1}' > "$DEST/ollama-models.txt"

Make executable and schedule:

chmod +x /opt/ai-stack/scripts/backup.sh
sudo crontab -e
0 3   * /opt/ai-stack/scripts/backup.sh >> /var/log/ai-stack-backup.log 2>&1

Off-site copy with rclone (recommended):

sudo apt install -y rclone
rclone config   # set up a remote, e.g. Backblaze B2
Add to the end of backup.sh:
rclone sync "$BACKUP_ROOT" b2:my-ai-stack-backups/ --transfers 4

Step 12: Monitoring

Stand up Uptime Kuma on the same (or a separate) VPS to watch the stack. See our dedicated guide How to Install Uptime Kuma on Ubuntu for the full walkthrough.

Configure monitors:

bash <(curl -Ss https://my-netdata.io/kickstart.sh) --non-interactive

Netdata auto-detects Docker and NVIDIA GPUs and exposes per-container CPU, RAM, IO, and GPU utilization out of the box.

Scaling Considerations

Vertical first. Adding vCPU and RAM to your existing VPS is almost always the cheapest scaling path. Llama 3.1 8B Q4 needs ~6 GB RAM for weights + 2–4 GB context; doubling RAM lets you hold 70B models quantized to Q4 (~40 GB). Contabo's CloudCore tiers let you resize in-place.

Horizontal for inference. Once a single Ollama can't keep up, run a second Ollama container (same network, different container name), add both to LiteLLM's model_list with a matching model_name, and LiteLLM will round-robin.

- model_name: llama3.1
  litellm_params: { model: ollama/llama3.1:8b, api_base: http://ollama:11434 }
model_name: llama3.1
  litellm_params: { model: ollama/llama3.1:8b, api_base: http://ollama-2:11434 }

Qdrant at scale. Qdrant on a single node happily holds tens of millions of vectors. Past that, enable Qdrant Cluster mode (2+ replicas) — see the Qdrant docs for the full compose example.

GPU migration. If you start on CPU and later move to a GPU host, migration is copy-the-volumes-and-boot:

# On old host:
docker run --rm -v ai-stack_ollama-data:/data -v $(pwd):/b alpine \
  tar czf /b/ollama-data.tar.gz -C /data .
scp to new host, restore into same-named volume, docker compose up -d

Multi-tenant. If different teams need isolation, give each their own Open WebUI workspace (built-in feature) and separate Qdrant collections. For hard isolation, run a second Open WebUI container pointed at the same Ollama but a different Qdrant collection prefix.

Optional Additions

LibreChat — a more feature-rich alternative frontend (plugins, agents, better multi-modal). Runs alongside Open WebUI, shares Ollama, and connects through LiteLLM. Add as a sixth service and route alt-chat.yourdomain.com to it.

Dify — a full AI-application platform with a visual workflow builder, RAG studio, and published-app endpoints. Heavier (Postgres + Redis + Weaviate required) but unbeatable for building internal AI apps without code.

SearXHG (SearXNG) — a privacy-first meta search engine. Wire it up to Open WebUI so the model can do live web search without calling Google's API.

searxng:
    image: searxng/searxng:latest
    container_name: searxng
    restart: unless-stopped
    environment:
      - SEARXNG_BASE_URL=https://search.yourdomain.com
    volumes:
      - ./configs/searxng:/etc/searxng
    networks:
      - ai-net

Then in Open WebUI → Settings → Web Search, pick searxng and set URL http://searxng:8080/search.

n8n — workflow automation with native Ollama + OpenAI nodes. Point n8n's OpenAI credentials at your LiteLLM URL and master key, and every n8n workflow gets your private models for free.

Continue.dev + Cursor — point these editor extensions at https://api.yourdomain.com/v1 with the LiteLLM master key and your dev team gets Copilot-equivalent autocomplete powered entirely by your VPS.

Troubleshooting

Open WebUI says "Ollama: no models found". DNS inside Docker: confirm OLLAMA_BASE_URL=http://ollama:11434 (container name, not localhost). Restart Open WebUI: docker compose up -d open-webui.

Chat streams but cuts off mid-response. nginx proxy_read_timeout too low or proxy_buffering on. The sample config sets both correctly; re-check /etc/nginx/conf.d/ai-stack.conf.

Qdrant 401/403 errors. API key mismatch. The key must appear in three places: .env, the Qdrant service env, and Open WebUI's QDRANT_API_KEY. Re-run docker compose up -d after any change.

Uploads fail with 413. client_max_body_size in nginx. We set 200M; bump if you need larger PDFs.

RAG retrieves irrelevant chunks. Three knobs: chunk size (try 800/100 for short factual docs, 1500/200 for narrative), top-k (5 is default; try 8), and the embedding model itself (swap nomic-embed-text for mxbai-embed-large — better English, larger vectors, slightly slower).

Ollama OOM killed. Your host ran out of RAM. Either reduce OLLAMA_MAX_LOADED_MODELS to 1, pick a smaller quant (e.g. llama3.1:8b-instruct-q4_K_S instead of Q5), or resize the VPS.

LiteLLM 401 "Invalid API key". The key in OPENAI_API_KEY (inside Open WebUI's env) must match LITELLM_MASTER_KEY exactly — same leading sk- prefix. Regenerate both if in doubt.

GPU not detected inside container. docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi must succeed on the host first. If it does but the stack still CPU-inferences, confirm the deploy.resources.reservations.devices block in the Ollama service is uncommented.

Docling timeouts on huge PDFs. Increase DOCLING_SERVE_MAX_SYNC_WAIT to 600 and the nginx timeouts to 900. For 500-page documents split first or use Docling's async mode.

FAQ

Can I run this without Docker? Yes — each component installs natively (Ollama has a one-line installer, Qdrant ships a binary, Open WebUI runs under uvicorn). Docker Compose just makes it reproducible and upgradeable.

Do I need a GPU? No. 7B–9B models run at 10–25 tokens/second on a modern 8-core CPU, which is fine for team chat. You want a GPU if you need 70B+ or sub-100-ms latency at high concurrency.

How many concurrent users can one VPS handle? With CPU inference on an 8-core host, plan for 3–5 concurrent active chats before queueing becomes noticeable. With a 24 GB GPU, 20–40 is realistic on 8B models.

How does this compare to just using ChatGPT Team? ChatGPT Team is $30/user/mo and caps at OpenAI's models. The self-hosted stack is flat $30–60/mo total, runs any open model, integrates via API everywhere, and never sends a byte to OpenAI. You lose frontier-model quality on hard coding tasks — mitigate by adding an API key to LiteLLM for hybrid routing.

Is my data really private? Yes, with caveats. Nothing leaves the VPS unless (a) you enable an external API in LiteLLM and route traffic to it, or (b) Open WebUI checks for updates (set WEBUI_CHECK_FOR_UPDATES=false to disable). Run sudo tcpdump -i any during a chat to verify zero outbound traffic if you are paranoid.

Can I put this behind my corporate SSO? Open WebUI supports OIDC (Google Workspace, Azure AD, Authentik, Keycloak). Set the OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, and OIDC_ISSUER_URL env vars and disable local signup.

How do I upgrade? docker compose pull && docker compose up -d. Before any major version bump run the backup script first. Open WebUI, LiteLLM, and Qdrant all publish clean upgrade notes with each release.

Will models update automatically? No. Ollama models are immutable once pulled. To update Llama 3.1 when a new point release drops: docker compose exec ollama ollama pull llama3.1:8b — Ollama diffs the layers and replaces only what changed.

Can I sell access to this? Absolutely — this is the stack powering many SaaS products. Point your app at https://api.yourdomain.com/v1 with a LiteLLM virtual key per customer, set per-key budgets, and bill on top.

Next Steps

You now own a complete, production-capable AI stack. Good follow-ups:

• Install Uptime Kuma to monitor it.
• Harden SSH on the host if you have not already.
• Set up automated off-site backups with rclone to Backblaze B2 or S3.
• Install n8n and wire it up to LiteLLM for no-code AI workflows.
• Fine-tune a custom model on your own data and load it straight into Ollama.