How to Install text-generation-webui (oobabooga) on Ubuntu 24.04

If Ollama is the Docker of local LLMs, text-generation-webui — better known by its GitHub handle oobabooga — is the Photoshop. It is the Swiss-army interface for running, chatting with, fine-tuning, and experimenting with every major open-weights large language model on the planet. Where most runtimes specialize in one format or one backend, oobabooga bolts together almost every serious inference library behind a single Gradio web UI, giving you GGUF, GPTQ, AWQ, EXL2, and vanilla HuggingFace Transformers all under one roof.

This guide walks you through installing text-generation-webui on Ubuntu 24.04, from system packages through GPU drivers, the one-click installer, manual Python setup, model downloads, the chat UI, the OpenAI-compatible API, production hardening with nginx, and a systemd service. By the end you will have a reproducible self-hosted LLM playground that can serve both humans (in a browser) and applications (via REST API) from the same process.

Skip the setup? Deploy a pre-configured GPU VPS with CUDA, cuDNN, and Python ready to go. Launch a GPU Server now and have oobabooga running in under 10 minutes.

Table of Contents

What is text-generation-webui?

text-generation-webui is an open-source, self-hostable web interface for running large language models locally. It was created by the developer known as oobabooga and has become the de facto standard for hobbyists, researchers, and power users who need to experiment with multiple LLMs, quantization formats, and inference backends without rebuilding their environment every time.

At its core the project is a thin Gradio wrapper around a "loader" abstraction: you pick a model, pick a loader that knows how to execute that model's format, and the UI handles everything else — chat history, sampling parameters, context windows, streaming, stop tokens, character cards, LoRAs, and extensions. Under the hood it bundles virtually every mainstream inference engine: Hugging Face Transformers, llama.cpp, ExLlamaV2, AutoGPTQ, AutoAWQ, HQQ, and more.

Think of oobabooga as a universal LLM cockpit. Ollama is great if you want a curated library of models and a simple ollama run. LM Studio is great if you want a polished desktop app. But if you want to download a random .gguf from Hugging Face, tweak the rope scaling, apply two LoRAs on top, switch between chat and notebook modes, expose an OpenAI-compatible API, and train a new LoRA — all in the same browser tab — text-generation-webui is the tool.

Supported Loaders and Model Formats

One of oobabooga's defining features is the number of inference backends it ships with. At time of writing, the project supports the following loaders:

• Transformers — The reference Hugging Face implementation. Loads any FP16/BF16 model in .safetensors or .bin format. Slowest but most compatible; useful for tiny models, CPU inference, and anything experimental that other loaders don't yet support.
• llama.cpp — The workhorse CPU/GPU loader for GGUF quantized models. Runs Llama, Mistral, Qwen, Gemma, Phi, and every derivative. Supports partial GPU offload via n_gpu_layers, making it the best choice when your VRAM is tight.
• ExLlamaV2 (exllamav2 / exllamav2_HF) — Blazing-fast GPU loader for EXL2 format models. Typically 2–3x faster than Transformers on the same hardware. Requires everything to fit in VRAM.
• AutoGPTQ — Loader for the older but still widely distributed GPTQ 4-bit quantization format. Many classic community finetunes are only available in GPTQ.
• AutoAWQ — Loader for AWQ (Activation-aware Weight Quantization), a 4-bit format that often produces better quality than GPTQ at the same bit-width.
• HQQ — Half-Quadratic Quantization. A newer on-the-fly quantizer that can run any HF model at 2/3/4/8 bits without a pre-quantized file.
• TensorRT-LLM (experimental) — NVIDIA's optimized inference runtime for maximum throughput on H100/A100 class hardware.

Why Use oobabooga Instead of Ollama or LM Studio?

Choose text-generation-webui when you need any of the following:

• Format freedom. You are not locked into a single runtime. GGUF, GPTQ, AWQ, EXL2, raw HF weights — all supported.
• Research-grade sampling controls. Temperature, top-p, top-k, min-p, typical-p, tail-free sampling, dynamic temperature, DRY, XTC, Mirostat, repetition penalty, frequency penalty, presence penalty — all exposed as sliders.
• LoRA stacking. Apply one or more LoRAs on top of a base model at load time.
• Training tab. Fine-tune LoRAs directly in the UI using QLoRA. Minimal code required.
• Extensions ecosystem. Long-term memory, TTS, STT, translation, web search, superbooga RAG, character cards, image generation hooks.
• Dual-purpose deployment. The same process serves a Gradio chat UI on port 7860 and an OpenAI-compatible API on port 5000 — feed both humans and agents from one GPU.
• Hackability. The entire codebase is Python with clean module boundaries. Write a custom extension in 50 lines.

Prerequisites

Before starting, make sure you have:

• Ubuntu 24.04 LTS with root or sudo access.
• At least 16 GB RAM (32 GB recommended for 13B models, 64 GB+ for 70B).
• 50 GB free disk space per model family. LLM weights are big — a single 70B model in 4-bit is ~40 GB.
• NVIDIA GPU strongly recommended. A 12 GB card runs 7B–13B comfortably, 24 GB runs 34B, 48 GB+ runs 70B. CPU-only works for GGUF but is slow.
• Python 3.11. The installer builds its own miniconda environment, so system Python does not matter much, but 3.11 is the tested version.
• Git, build tools, and curl.
• A Hugging Face account and access token if you want to download gated models (Llama, Gemma).

Step 1: System Preparation

SSH into your Ubuntu 24.04 box and bring everything up to date:

sudo apt update && sudo apt upgrade -y
sudo apt install -y \
    build-essential \
    git \
    curl \
    wget \
    ca-certificates \
    software-properties-common \
    python3-dev \
    python3-venv \
    python3-pip \
    ninja-build \
    pkg-config \
    libopenblas-dev \
    ufw

Create a dedicated non-root user to run the service. Running LLM software as root is a bad idea — these codebases pull in hundreds of packages, many of which are research-grade.

sudo adduser --disabled-password --gecos "" oobabooga
sudo usermod -aG sudo oobabooga
sudo su - oobabooga

Everything from here on runs as the oobabooga user.

Step 2: Install NVIDIA Drivers and CUDA

Skip this step if you are running CPU-only.

Check whether the kernel already sees your GPU:

lspci | grep -i nvidia

If a card is listed, install the proprietary driver. Ubuntu 24.04 ships driver 535 in the main repo; for newer cards (RTX 40 series, H100) use the graphics-drivers PPA:

sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt update
sudo ubuntu-drivers install
sudo reboot

After reboot, verify:

nvidia-smi

You should see your GPU, driver version, and CUDA version (e.g. 12.4). The oobabooga installer will download a matching CUDA toolkit into its own conda environment, so you don't need a system-wide CUDA install — only the driver.

Step 3: Automatic Installer (Recommended)

The project ships a one-click installer that creates a self-contained miniconda environment, installs PyTorch with the correct CUDA version, and pulls every loader. This is the easiest path and the one the maintainer officially supports.

cd ~
git clone https://github.com/oobabooga/text-generation-webui.git
cd text-generation-webui
./start_linux.sh

The script will prompt you to choose your GPU vendor:

What is your GPU?
A) NVIDIA
B) AMD (Linux/MacOS only)
C) Apple M Series
D) Intel Arc (IPEX)
N) None (I want to run models in CPU mode)

Pick A for NVIDIA on a VPS. The installer downloads miniconda, creates an installer_files/env directory, and installs roughly 5 GB of dependencies. Expect 5–15 minutes depending on your bandwidth.

When the installer finishes it launches the server and prints:

Running on local URL:  http://127.0.0.1:7860

Press Ctrl+C to stop. From now on, launch the server with:

./start_linux.sh

And update it with:

./update_wizard_linux.sh

Step 4: Manual Python Install (Alternative)

If you prefer explicit control — for example in a reproducible container image — install manually. This mirrors what the installer does but keeps everything visible.

cd ~
git clone https://github.com/oobabooga/text-generation-webui.git
cd text-generation-webui
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools
PyTorch with CUDA 12.1 (adjust for your driver)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Core requirements
pip install -r requirements/full/requirements.txt

The requirements/ directory contains variants: full for all loaders, cpu_only for CPU-only, noavx2 for old CPUs without AVX2. Pick the one that matches your hardware.

Launch manually:

python server.py

Step 5: Download a Model from Hugging Face

text-generation-webui includes a model downloader. Models live in the models/ subdirectory.

Option A — Use the built-in UI. Start the server, open the Model tab, paste a Hugging Face repo path (e.g. TheBloke/Mistral-7B-Instruct-v0.2-GGUF), optionally specify a single filename (e.g. mistral-7b-instruct-v0.2.Q4_K_M.gguf) to skip downloading every quant, and click Download.

Option B — Use the CLI downloader. Faster for scripting:

cd ~/text-generation-webui
python download-model.py TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
    --specific-file mistral-7b-instruct-v0.2.Q4_K_M.gguf

For gated repos (Meta Llama, Google Gemma), export your Hugging Face token first:

export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxx

Good starter models:

Step 6: Launch the Server with --listen

By default the Gradio server binds to 127.0.0.1, which is only accessible from inside the VPS. To reach it from your laptop, pass --listen:

./start_linux.sh --listen --listen-port 7860 --api --api-port 5000

Flag reference:

• --listen — bind to 0.0.0.0 instead of loopback.
• --listen-port 7860 — Gradio UI port.
• --api — enable the OpenAI-compatible HTTP API.
• --api-port 5000 — API port (default is 5000).
• --gradio-auth user:password — protect the UI with basic auth.
• --ssl-keyfile / --ssl-certfile — terminate HTTPS inside Gradio (we prefer nginx, see below).
• --verbose — log every prompt and response.

sudo ufw allow 22/tcp
sudo ufw allow 7860/tcp
sudo ufw enable

Browse to http://YOUR_SERVER_IP:7860 and you should see the oobabooga interface.

Step 7: Using the Chat UI

The Chat tab is where most users live. After downloading a model:

Chat mode supports three instruction templates:

• Chat — freeform roleplay using character cards.
• Chat-instruct — wraps your message in an instruction template before sending. Best for instruct-tuned models.
• Instruct — pure single-turn instruction mode.

Step 8: Notebook Mode and Parameters

Switch to the Notebook tab when you need raw completion behavior — no roleplay wrapper, no system prompt, just the model continuing your text. This is the mode researchers and prompt engineers use. The Default tab is similar but presents input and output in two side-by-side boxes.

The Parameters tab is the heart of the app. Key knobs:

• Temperature — 0 is deterministic, 1 is creative, 2 is chaos.
• Top P / Top K / Min P — nucleus and cutoff sampling. Min P (0.05) is the modern default.
• Repetition penalty — 1.1–1.2 for most models.
• DRY / XTC — newer anti-repetition samplers that dramatically improve long generations.
• Mirostat — adaptive sampling that targets a constant perplexity.
• Seed — set to a positive integer for reproducible outputs.
• Max new tokens — generation cap per response.
• Truncate the prompt up to this length — how much context to keep when the conversation exceeds the window.

Step 9: Extensions (LongTermMemory, coqui_tts, translate)

Extensions live in the extensions/ directory and are toggled in the Session tab or via --extensions on the command line.

LongTermMemory — persistent conversational memory backed by a local vector store. Remembers facts across sessions.

cd ~/text-generation-webui/extensions
git clone https://github.com/wawawario2/long_term_memory.git

Enable with --extensions long_term_memory.

coqui_tts — high-quality text-to-speech using the XTTS-v2 model. Clone a voice from a 10-second sample and have the bot speak its responses.

pip install TTS
Extension ships with the base repo; enable it:
./start_linux.sh --extensions coqui_tts

translate — Google Translate wrapper. Chat in any language; the extension translates your input to English before the model sees it and translates the response back.

./start_linux.sh --extensions google_translate

Other popular extensions:

• superboogav2 — local RAG using ChromaDB. Drop PDFs into a folder and the model can cite them.
• whisper_stt — voice input via OpenAI Whisper.
• sd_api_pictures — the bot generates images via a Stable Diffusion WebUI backend.
• openai — the built-in extension that enables the OpenAI-compatible API (auto-loaded with --api).

Step 10: OpenAI-Compatible API

Passing --api exposes an OpenAI v1-compatible HTTP server on port 5000. Any library that talks to OpenAI can be pointed at your oobabooga instance by changing the base_url.

curl example — chat completion:

curl http://localhost:5000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Explain RAG in one sentence."}
        ],
        "temperature": 0.7,
        "max_tokens": 200
    }'

Python example — the official OpenAI SDK:

from openai import OpenAI
client = OpenAI(
    base_url="http://YOUR_SERVER_IP:5000/v1",
    api_key="sk-not-used"  # any string works unless you set OPENEDAI_API_KEY
)
resp = client.chat.completions.create(
    model="mistral-7b-instruct",  # the currently loaded model
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Require an API key with the OPENEDAI_API_KEY environment variable:

export OPENEDAI_API_KEY=sk-your-long-random-string
./start_linux.sh --listen --api

Clients must then send Authorization: Bearer sk-your-long-random-string.

Endpoints available: /v1/models, /v1/chat/completions, /v1/completions, /v1/embeddings (if a supported embedding model is loaded), /v1/internal/model/load (oobabooga-specific, for hot-swapping models).

Step 11: Nginx Reverse Proxy with HTTP Basic Auth

Exposing port 7860 or 5000 directly to the internet is fine for testing but not for anything real. Put nginx in front, terminate TLS with Let's Encrypt, and add HTTP basic auth on top of the UI.

Install nginx and certbot:

sudo apt install -y nginx certbot python3-certbot-nginx apache2-utils

Create a password file:

sudo htpasswd -c /etc/nginx/.htpasswd admin

Write /etc/nginx/sites-available/oobabooga:

server {
    listen 80;
    server_name llm.example.com;
    # Redirect all HTTP to HTTPS (certbot will add this)
    location / {
        return 301 https://$host$request_uri;
    }
}
server {
    listen 443 ssl http2;
    server_name llm.example.com;
    # SSL config — certbot will fill in the paths
    ssl_certificate /etc/letsencrypt/live/llm.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
    client_max_body_size 100M;
    # Gradio UI — protected with basic auth
    location / {
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:7860;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s;
        proxy_buffering off;
    }
    # OpenAI-compatible API — NO basic auth; use OPENEDAI_API_KEY instead
    location /v1/ {
        proxy_pass http://127.0.0.1:5000/v1/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
        proxy_buffering off;
    }
}

Enable and certify:

sudo ln -s /etc/nginx/sites-available/oobabooga /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx -d llm.example.com

Now restrict the firewall: only 22, 80, and 443 should be open. Ports 7860 and 5000 stay on localhost.

sudo ufw delete allow 7860/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Step 12: Run as a Systemd Service

Manual launches do not survive reboots. Create a unit file so oobabooga starts on boot and auto-restarts on crash.

Write /etc/systemd/system/oobabooga.service:

[Unit]
Description=text-generation-webui (oobabooga)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=oobabooga
Group=oobabooga
WorkingDirectory=/home/oobabooga/text-generation-webui
Environment="OPENEDAI_API_KEY=sk-change-me-to-something-random"
Environment="HF_TOKEN=hf_your_huggingface_token"
ExecStart=/home/oobabooga/text-generation-webui/start_linux.sh \
    --listen \
    --listen-port 7860 \
    --api \
    --api-port 5000 \
    --model mistral-7b-instruct-v0.2.Q4_K_M.gguf \
    --loader llama.cpp \
    --n-gpu-layers 99 \
    --n_ctx 8192
Restart=on-failure
RestartSec=10
Allow long model-load times before systemd gives up
TimeoutStartSec=600
GPU access
DeviceAllow=/dev/nvidia* rwm
[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable oobabooga
sudo systemctl start oobabooga
sudo systemctl status oobabooga

Tail the logs:

journalctl -u oobabooga -f

Multi-GPU Setup

If the VPS has multiple GPUs, oobabooga can spread a model across them.

For Transformers / AutoGPTQ / AutoAWQ — use the gpu-memory slider per GPU in the Model tab, or pass comma-separated values:

./start_linux.sh --gpu-memory 20 20 --cpu-memory 32

This caps each GPU at 20 GiB and lets the remainder spill to system RAM.

For ExLlamaV2 — use gpu-split:

./start_linux.sh --loader exllamav2 --gpu-split "17,24"

The numbers are the per-GPU VRAM budget in GiB.

For llama.cpp — tensor split:

./start_linux.sh --loader llama.cpp \
    --tensor_split 0.5,0.5 \
    --n-gpu-layers 99

Evenly splits the GGUF across two cards. Works for arbitrary counts.

Pin the visible devices explicitly to avoid fighting with other processes:

CUDA_VISIBLE_DEVICES=0,1 ./start_linux.sh ...

Training and LoRA Tab Overview

The Training tab lets you fine-tune a LoRA adapter directly in the UI. It uses QLoRA (4-bit quantized base weights + trainable LoRA on top), which makes it possible to fine-tune a 13B model on a single 24 GB card.

Workflow:

For serious training, drive it from the command line with the --lora-dir argument or the underlying PEFT library. The UI is best for experimentation and small datasets (<100k examples).

Troubleshooting

CUDA out of memory — the model + context does not fit in VRAM. Pick a smaller quant (Q4_K_M instead of Q6_K), lower n_ctx, lower n-gpu-layers and spill to CPU, or upgrade to a bigger GPU plan.

RuntimeError: CUDA error: no kernel image is available for execution on the device — PyTorch was built for a different compute capability than your card. Reinstall with the correct CUDA wheel:

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

ImportError: cannot import name 'xxx' from 'transformers' — version drift between loaders and Transformers. Re-run ./update_wizard_linux.sh, or in manual installs pip install -r requirements/full/requirements.txt --upgrade.

Model loads but generates gibberish — wrong prompt template. Set the Instruction template in the Parameters → Instruction template tab to match the model (e.g. Llama-v3, Mistral, ChatML). For GGUF files that ship a chat template in metadata, check the Use chat template from the model metadata box.

llama.cpp: failed to allocate buffer — you offloaded more layers than fit in VRAM. Reduce n-gpu-layers.

Gradio UI loads blank, no errors — a browser cache issue after updates. Hard refresh with Ctrl+Shift+R, or append ?__theme=dark to the URL.

Extension fails to load — check journalctl -u oobabooga or the terminal output. Most extension failures are missing pip dependencies; read the extension's requirements.txt and pip install into the env:

source ~/text-generation-webui/installer_files/env/bin/activate
pip install -r extensions/long_term_memory/requirements.txt

Slow generation on GPU — check nvidia-smi during generation. If GPU-Util is low, the model is being CPU-bottlenecked: try a GPU-native loader (ExLlamaV2 for EXL2, AutoAWQ for AWQ) instead of llama.cpp.

FAQ

Q: Can I run oobabooga without a GPU? Yes — choose N in the installer or use --cpu. Stick to GGUF models at Q4_K_M or smaller, and expect 1–5 tokens/sec on a modern CPU depending on core count and memory bandwidth.

Q: Which loader is fastest? On NVIDIA GPUs, ExLlamaV2 is almost always the fastest when the model fits in VRAM. llama.cpp wins when you need CPU offload.

Q: How does the OpenAI-compatible API compare to the real OpenAI API? Functionally equivalent for chat and completions. Embeddings work if you load a supported embedding model. Function calling / tools work for models that were instruction-tuned for it (Llama 3.1, Hermes, Qwen2.5).

Q: Can I run multiple models simultaneously? Not in a single oobabooga process — it holds one model at a time. For multi-model serving, run multiple instances on different ports, or use a dedicated multi-model server like vLLM or Aphrodite Engine.

Q: Does it support vision models (LLaVA, Llama 3.2 Vision)? Partial — LLaVA is supported via the multimodal extension. Newer vision models are a moving target; check the GitHub issue tracker for current status.

Q: Is it safe to expose the API to the public internet? Only with an API key set (OPENEDAI_API_KEY), HTTPS, and ideally a rate limiter (nginx limit_req). LLM endpoints are expensive to abuse — a single malicious prompt can burn hours of GPU time.

Q: Can I use oobabooga with LangChain / LlamaIndex / CrewAI? Yes — point any OpenAI-compatible client at http://YOUR_SERVER:5000/v1 and pass any string as the API key (or your real key if set).

Q: How much disk space should I budget? The repo + env is ~10 GB. A 7B GGUF is 4–8 GB. A 70B GGUF is 35–45 GB. Budget 100 GB minimum, 500 GB if you want a real model library.

Next Steps

You now have a production-grade local LLM stack with a Gradio UI, an OpenAI-compatible API, HTTPS, basic auth, and a systemd service. Useful things to do next:

• Integrate into your editor. Install Continue.dev in VS Code and point it at http://YOUR_SERVER/v1 for code completion and chat.
• Build a RAG pipeline. Drop the superboogav2 extension in, feed it PDFs, and the bot will cite your documents.
• Fine-tune a LoRA on your company's support tickets or internal docs for a bespoke assistant.
• Chain multiple backends. Run oobabooga on one GPU for experimentation and vLLM on another for high-throughput production serving — both speak the same OpenAI API.
• Monitor GPU usage with nvtop and Netdata to right-size your VPS.