How to Install Whisper on Ubuntu 24.04 — Self-Hosted Speech-to-Text on Your VPS

Audio and video content drive podcasts, meetings, support calls, lectures, and customer interviews, but turning that audio into searchable, actionable text usually means shipping sensitive recordings to a third-party API and paying per minute. This guide walks you through installing OpenAI Whisper on an Ubuntu 24.04 VPS, so you own the transcripts, the models, and the infrastructure end-to-end. By the end you will have a production-ready speech-to-text service that transcribes audio in 99 languages, generates SRT/VTT subtitles, and exposes a clean FastAPI HTTP endpoint behind Nginx.

Skip the setup? Deploy Whisper in one click with our pre-configured AI Transcription image. Launch a Whisper-ready VPS now and start transcribing in under 60 seconds.

Table of Contents

What is Whisper?

Whisper is an open-source automatic speech recognition (ASR) model released by OpenAI in September 2022 and permissively MIT-licensed. It was trained on 680,000 hours of multilingual, multitask supervised audio — orders of magnitude more than most predecessors. That scale is why Whisper is robust to accents, background noise, music, technical jargon, and low-quality phone recordings that would trip up older recognizers.

Whisper is a family of encoder-decoder transformer checkpoints. Each checkpoint performs several tasks from the same weights: English transcription, multilingual transcription in 99 languages, direct translation of non-English audio into English, spoken language identification, and voice activity detection. You pass a flag to switch tasks — no model swap required. It is the same model family that powers OpenAI's paid whisper-1 API, but the weights are free to download and run on your own hardware.

For VPS operators, Whisper has no telemetry, no license server, and no runtime fee. Once the weights are on disk, the service runs entirely offline.

Whisper Model Sizes

Whisper ships in six sizes. Picking the right one is the single biggest decision you will make during setup, because it determines both accuracy and VPS specs.

The .en suffix variants (tiny.en, base.en, small.en, medium.en) are English-only checkpoints. They are slightly more accurate on English than their multilingual counterparts and use the same RAM, so if you know you will never transcribe another language, use them.

Why Self-Host Whisper?

The OpenAI Whisper API costs $0.006 per minute of audio. Run the math: a support team recording 8 hours of calls a day, 5 days a week, hits ~$60 a month; a 100-host podcast network pushing 1,000 hours a month hits $360 a month. Self-hosting on a VPS that costs $20-50 per month breaks even almost immediately.

Privacy and compliance is the other decisive factor. Medical consultations, legal depositions, HR investigations, and customer support calls often cannot legally leave your infrastructure under HIPAA, GDPR Article 28, or contractual confidentiality. Running Whisper locally keeps every byte of audio inside a VPS you control.

Batch processing without rate limits matters if you have a back catalog. Transcribing 500 hours of archived podcasts against a hosted API means babysitting rate limits and spend alerts. On your own VPS you queue the jobs and let them run.

Unlimited usage removes product decisions. You no longer need to decide which users get transcription or how many minutes per month they are allowed. The marginal cost of another minute is CPU cycles you already paid for.

Finally, customization: fine-tune Whisper on your own vocabulary, patch in custom initial prompts, chain it with speaker diarization models like pyannote, or quantize it to fit on a smaller VPS. None of that is possible against a closed API.

Prerequisites

Before you start you need:

• An Ubuntu 24.04 VPS with at least 4 vCPU and 8 GB RAM for small / turbo, or 8 vCPU / 16 GB for large-v3. A GPU is optional but makes turbo and large-v3 dramatically faster; see Method 3.
• Root or sudo access over SSH.
• 30 GB of free disk (models + ffmpeg cache + a few hours of audio).
• A non-root user you will run the service as (we use whisper below).
• Basic familiarity with the Linux command line.

Step 1: Update the System

SSH in and bring the base system current before installing anything else.

ssh root@your-vps-ip
apt update && apt upgrade -y
apt install -y build-essential git curl wget ffmpeg \
    python3 python3-pip python3-venv python3-dev

ffmpeg is mandatory. Whisper uses it internally to decode every audio format (MP3, M4A, FLAC, OGG, WAV, MP4, MKV, WebM) into the 16 kHz mono float32 tensors the model expects. Without it, nothing works.

Create a dedicated service user so Whisper never runs as root:

adduser --disabled-password --gecos "" whisper
usermod -aG sudo whisper
su - whisper

All remaining commands run as whisper.

Method 1: Install openai-whisper (Python package)

This is the official reference implementation from OpenAI. It is the easiest to install, the easiest to debug, and the baseline every other port is measured against.

1.1 Create a Python virtual environment

mkdir -p ~/whisper && cd ~/whisper
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip setuptools wheel

1.2 Install Whisper and PyTorch

pip install -U openai-whisper

On CPU-only VPS this pulls the CPU build of PyTorch automatically. If you have an NVIDIA GPU and have already installed the CUDA toolkit, install the CUDA build of torch first:

pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -U openai-whisper

1.3 Verify the installation

whisper --help
python -c "import whisper; print(whisper.available_models())"

You should see the full list: ['tiny', 'tiny.en', 'base', 'base.en', 'small', 'small.en', 'medium', 'medium.en', 'large-v1', 'large-v2', 'large-v3', 'large', 'large-v3-turbo', 'turbo'].

1.4 First transcription

Grab a sample audio file and test end-to-end:

wget https://github.com/openai/whisper/raw/main/tests/jfk.flac
whisper jfk.flac --model turbo --output_format txt
cat jfk.txt

The first run downloads the model to ~/.cache/whisper/ (roughly 1.6 GB for turbo). Subsequent runs use the cached copy.

Method 2: Install whisper.cpp (faster CPU)

If you do not have a GPU, whisper.cpp from the author of llama.cpp is almost always the right choice. It is a pure C/C++ port with zero Python dependencies, aggressive quantization (Q5, Q8, Q4), ARM NEON and AVX2 acceleration, and an OpenVINO backend for Intel CPUs. On the same CPU, whisper.cpp is typically 2-4x faster than the Python implementation.

2.1 Clone and build

cd ~
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make -j$(nproc)

2.2 Download a model

whisper.cpp uses GGML-format models. A helper script converts or downloads them:

bash ./models/download-ggml-model.sh turbo
or
bash ./models/download-ggml-model.sh large-v3-q5_0

The q5_0 suffix is a 5-bit quantization that cuts the model to ~1 GB with minimal accuracy loss.

2.3 Transcribe

./build/bin/whisper-cli -m models/ggml-turbo.bin -f samples/jfk.wav

For long files, use -t $(nproc) to parallelize across all cores and -of output --output-txt --output-srt to dump results.

Method 3: Install faster-whisper (GPU-optimized)

faster-whisper is a reimplementation on top of CTranslate2, a highly optimized inference engine. On GPU it is roughly 4x faster than the reference implementation at the same accuracy, and it uses significantly less VRAM (int8 quantization can run large-v3 on 4 GB of VRAM). It is the right choice for any production API.

3.1 Install CUDA (GPU VPS only)

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-4 libcudnn9-cuda-12
nvidia-smi

3.2 Install faster-whisper

cd ~/whisper
source venv/bin/activate
pip install faster-whisper

3.3 Transcribe in Python

from faster_whisper import WhisperModel
model = WhisperModel("turbo", device="cuda", compute_type="float16")
For CPU: WhisperModel("turbo", device="cpu", compute_type="int8")
segments, info = model.transcribe("audio.mp3", beam_size=5)
print(f"Detected language: {info.language} (p={info.language_probability:.2f})")
for seg in segments:
    print(f"[{seg.start:.2f} -> {seg.end:.2f}] {seg.text}")

compute_type="int8" on CPU, "float16" on consumer GPUs, "bfloat16" on newer Ada / Hopper cards.

Method 4: Install Whisper WebUI (user-friendly)

If non-technical teammates need to drop files in a browser and get transcripts back, install the Whisper WebUI. It wraps faster-whisper in a Gradio interface with drag-and-drop upload, language picker, subtitle downloads, and YouTube URL input.

cd ~
git clone https://github.com/jhj0517/Whisper-WebUI.git
cd Whisper-WebUI
pip install -r requirements.txt
python app.py --server_name 0.0.0.0 --server_port 7860

Visit http://your-vps-ip:7860 and transcribe interactively. Put it behind the Nginx config in the section below so your team can access it at https://whisper.yourdomain.com.

Transcribing Audio Files via CLI

The whisper CLI (from Method 1) has the most flags and is the fastest way to transcribe a directory of files.

Basic transcription with defaults:

whisper meeting.mp3

Pin the model, language, and output formats:

whisper meeting.mp3 \
    --model turbo \
    --language English \
    --output_format all \
    --output_dir transcripts/

--output_format all writes .txt, .srt, .vtt, .tsv, and .json simultaneously. The JSON file contains word-level timestamps, segment confidence scores, and token probabilities and is the right format to feed into a downstream database.

Useful flags for production:

• --device cuda — force GPU
• --fp16 True — half precision (default on GPU, faster)
• --temperature 0 — deterministic output (no sampling)
• --initial_prompt "Acme Corp, Kubernetes, Stripe" — bias the decoder toward your vocabulary
• --condition_on_previous_text False — prevents runaway hallucinations on long silences

Building a Transcription API with FastAPI

The CLI is great for one-offs; for apps and cron jobs you want an HTTP endpoint. The script below exposes a clean /transcribe endpoint using faster-whisper, accepts uploaded audio/video, and returns JSON with segments and optional SRT/VTT.

Install the server dependencies:

cd ~/whisper
source venv/bin/activate
pip install fastapi "uvicorn[standard]" python-multipart faster-whisper

Save this as ~/whisper/server.py:

"""
Self-hosted Whisper transcription API.
Runs on vps-server.host — https://vps-server.host
"""
import os
import tempfile
import uuid
from pathlib import Path
from typing import Optional
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
from fastapi.responses import JSONResponse, PlainTextResponse
from faster_whisper import WhisperModel
MODEL_SIZE = os.getenv("WHISPER_MODEL", "turbo")
DEVICE = os.getenv("WHISPER_DEVICE", "cpu")        # "cuda" on GPU VPS
COMPUTE_TYPE = os.getenv("WHISPER_COMPUTE", "int8")  # "float16" on GPU
MAX_UPLOAD_MB = int(os.getenv("WHISPER_MAX_MB", "500"))
API_KEY = os.getenv("WHISPER_API_KEY", "")         # set in systemd unit
print(f"Loading model {MODEL_SIZE} on {DEVICE} ({COMPUTE_TYPE})...")
model = WhisperModel(MODEL_SIZE, device=DEVICE, compute_type=COMPUTE_TYPE)
print("Model ready.")
app = FastAPI(title="Whisper API", version="1.0")
def _check_key(authorization: Optional[str]) -> None:
    if not API_KEY:
        return
    expected = f"Bearer {API_KEY}"
    if authorization != expected:
        raise HTTPException(status_code=401, detail="invalid api key")
def _format_timestamp(seconds: float, vtt: bool = False) -> str:
    h = int(seconds // 3600)
    m = int((seconds % 3600) // 60)
    s = seconds % 60
    sep = "." if vtt else ","
    return f"{h:02d}:{m:02d}:{s:06.3f}".replace(".", sep, 1)
def _segments_to_srt(segments) -> str:
    lines = []
    for i, seg in enumerate(segments, 1):
        lines.append(str(i))
        lines.append(f"{_format_timestamp(seg['start'])} --> {_format_timestamp(seg['end'])}")
        lines.append(seg["text"].strip())
        lines.append("")
    return "\n".join(lines)
def _segments_to_vtt(segments) -> str:
    lines = ["WEBVTT", ""]
    for seg in segments:
        lines.append(
            f"{_format_timestamp(seg['start'], vtt=True)} --> "
            f"{_format_timestamp(seg['end'], vtt=True)}"
        )
        lines.append(seg["text"].strip())
        lines.append("")
    return "\n".join(lines)
@app.get("/health")
def health():
    return {"status": "ok", "model": MODEL_SIZE, "device": DEVICE}
@app.post("/transcribe")
async def transcribe(
    file: UploadFile = File(...),
    language: Optional[str] = Form(None),
    task: str = Form("transcribe"),        # or "translate"
    output: str = Form("json"),            # json | text | srt | vtt
    initial_prompt: Optional[str] = Form(None),
    authorization: Optional[str] = None,
):
    _check_key(authorization)
    if task not in ("transcribe", "translate"):
        raise HTTPException(400, "task must be 'transcribe' or 'translate'")
    if output not in ("json", "text", "srt", "vtt"):
        raise HTTPException(400, "output must be json|text|srt|vtt")
    suffix = Path(file.filename or "audio").suffix or ".bin"
    tmp = Path(tempfile.gettempdir()) / f"whisper-{uuid.uuid4().hex}{suffix}"
    size_bytes = 0
    try:
        with tmp.open("wb") as f:
            while chunk := await file.read(1024 * 1024):
                size_bytes += len(chunk)
                if size_bytes > MAX_UPLOAD_MB  1024  1024:
                    raise HTTPException(413, f"file exceeds {MAX_UPLOAD_MB} MB")
                f.write(chunk)
        segments_iter, info = model.transcribe(
            str(tmp),
            language=language,
            task=task,
            initial_prompt=initial_prompt,
            beam_size=5,
            vad_filter=True,
            word_timestamps=(output == "json"),
        )
        segments = [
            {
                "id": i,
                "start": round(s.start, 3),
                "end": round(s.end, 3),
                "text": s.text,
                "words": (
                    [{"w": w.word, "s": round(w.start, 3), "e": round(w.end, 3)}
                     for w in (s.words or [])]
                    if output == "json" else None
                ),
            }
            for i, s in enumerate(segments_iter)
        ]
        full_text = " ".join(s["text"].strip() for s in segments).strip()
        if output == "text":
            return PlainTextResponse(full_text)
        if output == "srt":
            return PlainTextResponse(_segments_to_srt(segments), media_type="text/plain")
        if output == "vtt":
            return PlainTextResponse(_segments_to_vtt(segments), media_type="text/vtt")
        return JSONResponse({
            "language": info.language,
            "language_probability": round(info.language_probability, 3),
            "duration": round(info.duration, 2),
            "text": full_text,
            "segments": segments,
        })
    finally:
        tmp.unlink(missing_ok=True)

Launch it:

export WHISPER_API_KEY="$(openssl rand -hex 32)"
uvicorn server:app --host 0.0.0.0 --port 9000 --workers 1

Test:

curl -X POST http://localhost:9000/transcribe \
    -H "Authorization: Bearer $WHISPER_API_KEY" \
    -F "[email protected]" \
    -F "output=srt"

One worker is usually correct: Whisper is CPU- or GPU-bound and extra workers just fight each other for the same cores. Use a queue (Redis RQ, Celery) in front of the API if you need concurrency.

Real-Time (Streaming) Transcription

Whisper was trained on 30-second chunks, so true zero-latency streaming is not its native strength, but projects like WhisperLive layer a WebSocket streaming interface on top:

pip install whisper-live
python -m whisper_live.server --port 9090 --backend faster_whisper --model turbo

Expect 1-3 seconds of latency on CPU, under a second on GPU.

Multilingual Transcription & Translation

Whisper covers 99 languages out of the box. Two task modes matter:

• transcribe — output in the same language as the audio.
• translate — output in English, regardless of the source language.

whisper podcast-es.mp3 --model turbo --task translate --output_format srt

For best results on non-English audio, bias the decoder with an --initial_prompt in the target language that contains any proper nouns the model might mangle (brand names, people, places). Keep the prompt under 200 tokens.

Language auto-detection is accurate but not free; if you already know the language, pass --language fr (ISO 639-1 codes) to shave a second off and avoid the occasional misdetect on short clips.

Generating Subtitles (SRT / VTT)

Both SRT and WebVTT are first-class outputs. The CLI emits them directly:

whisper lecture.mp4 --model turbo --output_format srt
whisper lecture.mp4 --model turbo --output_format vtt

Tune segment length for readable subtitles:

whisper lecture.mp4 \
    --model turbo \
    --output_format srt \
    --max_line_width 42 \
    --max_line_count 2 \
    --word_timestamps True

Word-level timestamps let Whisper break lines at natural phrase boundaries instead of arbitrary 30-second chunks. The resulting SRT is almost publication-ready. Burn it into a video with:

ffmpeg -i lecture.mp4 -vf "subtitles=lecture.srt" -c:a copy lecture-subbed.mp4

Running Whisper as a systemd Service

Keep the API running across reboots and log to the journal. Switch to root briefly:

sudo tee /etc/systemd/system/whisper.service > /dev/null <<'EOF'
[Unit]
Description=Whisper transcription API
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=whisper
Group=whisper
WorkingDirectory=/home/whisper/whisper
Environment="PATH=/home/whisper/whisper/venv/bin:/usr/bin"
Environment="WHISPER_MODEL=turbo"
Environment="WHISPER_DEVICE=cpu"
Environment="WHISPER_COMPUTE=int8"
Environment="WHISPER_MAX_MB=500"
EnvironmentFile=/etc/whisper.env
ExecStart=/home/whisper/whisper/venv/bin/uvicorn server:app \
    --host 127.0.0.1 --port 9000 --workers 1 --timeout-keep-alive 300
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/whisper/whisper /tmp
[Install]
WantedBy=multi-user.target
EOF
echo "WHISPER_API_KEY=$(openssl rand -hex 32)" | sudo tee /etc/whisper.env
sudo chmod 600 /etc/whisper.env
sudo systemctl daemon-reload
sudo systemctl enable --now whisper
sudo systemctl status whisper
journalctl -u whisper -f

The service now listens on 127.0.0.1:9000 only. Public traffic goes through Nginx.

Nginx Reverse Proxy with Large Upload Support

Audio and video files are big. The default Nginx 1 MB upload ceiling and the default 60-second proxy timeout will both bite you immediately. Raise both.

sudo apt install -y nginx certbot python3-certbot-nginx
sudo tee /etc/nginx/sites-available/whisper > /dev/null <<'EOF'
server {
    listen 80;
    server_name whisper.yourdomain.com;
    # Allow up to 2 GB uploads — adjust to match your audio lengths.
    client_max_body_size 2048M;
    client_body_buffer_size 1M;
    client_body_timeout 600s;
    location / {
        proxy_pass http://127.0.0.1:9000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Give the model time to chew through long recordings.
        proxy_connect_timeout 60s;
        proxy_send_timeout    600s;
        proxy_read_timeout    600s;
        send_timeout          600s;
        # Stream uploads to the app rather than buffering to disk first.
        proxy_request_buffering off;
        proxy_buffering off;
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/whisper /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx -d whisper.yourdomain.com

client_max_body_size is the single most common reason uploads 413 when people follow this tutorial. Set it generously. proxy_request_buffering off streams the upload straight to FastAPI instead of spooling the whole file to Nginx's /var/cache/nginx/client_temp first, which saves disk IO on 1 GB+ uploads.

Security Hardening

sudo ufw allow 22/tcp && sudo ufw allow 80/tcp && sudo ufw allow 443/tcp
   sudo ufw enable

Troubleshooting

ffmpeg: command not found — install ffmpeg: sudo apt install -y ffmpeg. Whisper shells out to it for every file.

RuntimeError: CUDA out of memory — drop to a smaller model, switch to compute_type="int8_float16", or add --fp16 True. large-v3 needs roughly 10 GB of VRAM in fp16.

Transcription loops the same sentence forever — a known Whisper hallucination on silence or music. Enable VAD (vad_filter=True in faster-whisper) and set --condition_on_previous_text False.

413 Request Entity Too Large from Nginx — bump client_max_body_size and reload. Both Nginx and FastAPI limits must be raised.

Uploads hang then timeout — increase proxy_read_timeout to match your longest expected job, and set --timeout-keep-alive 300 on uvicorn.

First request is very slow — the model loads into RAM on first use. Pre-warm by calling /health from the systemd unit, or pin the model in memory with a dummy transcribe on startup.

Transcripts are wrong for technical terms — use --initial_prompt "Kubernetes, Prometheus, Grafana, Terraform" to bias the decoder toward your vocabulary.

Wrong language detected — pass --language en explicitly for short clips under 30 seconds, where auto-detect is unreliable.

FAQ

Does Whisper work offline? Yes. After the first model download, no network access is required.

How many concurrent requests can one VPS handle? One at a time per worker on CPU. For concurrency, run a queue (Celery, RQ) in front of the API and scale workers across multiple VPS.

Can I fine-tune Whisper? Yes, using Hugging Face transformers and the openai/whisper-* checkpoints. Expect to need a GPU VPS and a few hundred labeled clips.

Does it do speaker diarization? Not natively. Chain Whisper with pyannote.audio — diarize first, then transcribe each speaker's clips.

Is Whisper suitable for medical or legal transcription? Yes for drafts, with a human review pass. No ASR is 100% accurate, and Whisper occasionally hallucinates on silence.

How do I transcribe YouTube videos? Use yt-dlp to download the audio, then feed it to Whisper:

yt-dlp -x --audio-format mp3 -o "video.%(ext)s" https://youtu.be/XXXX
whisper video.mp3 --model turbo

Large-v3 vs turbo? turbo is ~8x faster with comparable accuracy on major languages. Use large-v3 only for rare languages or extremely noisy audio.

Next Steps

You now have a production-ready, self-hosted speech-to-text service on your VPS. Natural next builds:

• Speaker diarization with pyannote.audio to produce "Alice: ... / Bob: ..." transcripts.
• Chain with an LLM running on Ollama to auto-summarize meetings and extract action items.
• Add a queue with Redis + RQ so long recordings process asynchronously with webhook callbacks.
• Index transcripts into Meilisearch or Typesense for full-text search across your audio archive.
• Drop a React UI on top of the API so non-technical teammates can upload files in a browser.

Ready to deploy? Pick a CloudCore Business VPS and have your transcription pipeline running today.