How to Install LocalAI on Ubuntu 24.04 — Your Own OpenAI-Compatible AI Server

LocalAI is the most complete OpenAI-compatible inference server you can self-host. Where Ollama focuses on LLM chat, LocalAI covers the full OpenAI API surface: chat completions, embeddings, image generation, text-to-speech, speech-to-text, vision, and function calling — all through the same endpoints your existing code already speaks. Drop in a LocalAI URL in place of api.openai.com, and your app keeps working without a single code change.

This tutorial walks you through a production install of LocalAI on Ubuntu 24.04 using Docker and Docker Compose, covers both CPU-only and GPU-accelerated variants, and shows how to pull models from the built-in gallery (Llama 3, Mistral, Phi, Gemma), call the API, generate images, and stream speech.

Skip the setup? Our CloudCore Business VPS comes with Docker pre-installed and NVMe storage tuned for AI workloads — deploy in under 60 seconds.

Table of Contents

What is LocalAI?

LocalAI is an open-source, self-hosted inference server that exposes a fully OpenAI-compatible REST API on top of multiple inference backends — llama.cpp, whisper.cpp, stable-diffusion.cpp, bark, piper, vLLM, Transformers, Diffusers, and more. The project is maintained by Ettore Di Giacinto (mudler) and a broad contributor community, and it has become the de facto choice for teams who want a single server that can replace several OpenAI endpoints at once.

The feature set is deliberately wide:

• Chat completions (/v1/chat/completions) with streaming, function calling, tools, and JSON mode
• Text completions (/v1/completions) for legacy clients
• Embeddings (/v1/embeddings) using sentence-transformers, BERT, or bge models
• Image generation (/v1/images/generations) via Stable Diffusion, SDXL, and FLUX
• Text-to-speech (/v1/audio/speech) with Piper, Bark, and Coqui voices
• Speech-to-text (/v1/audio/transcriptions) via Whisper
• Vision / multimodal chat with LLaVA, BakLLaVA, and Llama 3.2 Vision
• Reranking (/v1/rerank) for RAG pipelines
• Model gallery with one-click install of 200+ curated model presets

LocalAI vs Ollama vs vLLM vs llama.cpp

Choosing the right inference server matters. Here is how LocalAI compares to the main alternatives:

When to pick Ollama: you only need LLM chat and embeddings, and you want the lightest-weight developer experience. See our Ollama install guide.

When to pick vLLM: you have one or more dedicated GPUs, you need maximum tokens-per-second throughput, and you are serving dozens of concurrent users. See our vLLM install guide.

When to pick llama.cpp: you are embedding inference inside another application or need the absolute smallest binary footprint.

Prerequisites

Before you begin, you need:

• A VPS running Ubuntu 24.04 LTS with root or sudo access
• SSH access to your server
• At least 16 GB of RAM for comfortably running 7B-parameter models alongside LocalAI's stack (24 GB+ strongly recommended if you want embeddings + chat + image generation loaded simultaneously)
• At least 100 GB of free disk space — LocalAI models plus Docker images grow quickly once you add a Stable Diffusion checkpoint (~7 GB) and a Whisper model (~3 GB)
• A domain name pointed at your VPS if you plan to expose the API externally with SSL

Recommended Plan: CloudCore Business

LocalAI's superpower is running multiple modalities at once — LLM + embeddings + image + speech — which is memory-hungry. We recommend the CloudCore Business plan:

- 8 vCPU cores

- 24 GB RAM

- 200 GB NVMe SSD

- Unmetered bandwidth

- From EUR 29.99/month

For GPU-accelerated image generation and fast LLM inference at scale, the Business plan also unlocks our GPU VPS add-on tier with NVIDIA A30 / A40 / L40S options. Starter VPS plans (4 GB / 8 GB RAM) can run LocalAI but only with one small model loaded at a time — not enough headroom for the full stack.

Connect to your server:

ssh root@your-server-ip

Step 1: Update System Packages

Refresh package indexes and apply security updates before installing anything.

sudo apt update && sudo apt upgrade -y

Install a few helpers you will need throughout this guide:

sudo apt install -y ca-certificates curl gnupg git htop

If the kernel was upgraded, reboot:

sudo reboot

Step 2: Install Docker and Docker Compose

LocalAI ships as Docker images that bundle the correct backend binaries for each modality. Installing via Docker avoids a long chain of C++ and Python build dependencies.

Add Docker's official GPG key and repository:

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install the engine and the Compose plugin:

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Add your user to the docker group so you can run containers without sudo:

sudo usermod -aG docker $USER
newgrp docker

Verify the install:

docker --version
docker compose version

Expected output:

Docker version 27.3.1, build ce12230
Docker Compose version v2.29.7

Step 3: Choose Your LocalAI Variant (CPU vs GPU)

LocalAI publishes several pre-built images. Pick the one that matches your hardware:

• Just exploring / CPU-only VPS: latest-aio-cpu — everything works out of the box.
• Production with modern NVIDIA GPU: latest-gpu-nvidia-cuda-12 — lean image, install only the models you need.
• AMD GPU: latest-gpu-hipblas.

Step 4: Install NVIDIA Container Toolkit (GPU Only)

Skip this section if you are running CPU-only.

Confirm the GPU is visible to the host:

sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

After reboot:

nvidia-smi

You should see your card, driver version, and CUDA version. Next, install the NVIDIA Container Toolkit so Docker can pass the GPU into containers:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify the toolkit works:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

You should see the same nvidia-smi output from inside the container.

Step 5: Deploy LocalAI with Docker Compose

Create a project directory:

sudo mkdir -p /opt/localai && cd /opt/localai
sudo chown $USER:$USER /opt/localai
mkdir -p models images backend-data

Create docker-compose.yml:

nano docker-compose.yml

CPU variant:

services:
  localai:
    image: localai/localai:latest-aio-cpu
    container_name: localai
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - DEBUG=false
      - THREADS=6
      - CONTEXT_SIZE=4096
      - MODELS_PATH=/build/models
      - GALLERIES=[{"name":"localai","url":"github:mudler/LocalAI/gallery/index.yaml@master"}]
      - LOCALAI_API_KEY=${LOCALAI_API_KEY}
      - LOCALAI_WATCHDOG_IDLE=true
      - LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m
    volumes:
      - ./models:/build/models
      - ./images:/tmp/generated/images
      - ./backend-data:/build/backend-data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20s
      retries: 5

GPU variant (replace the services.localai block):

services:
  localai:
    image: localai/localai:latest-gpu-nvidia-cuda-12
    container_name: localai
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - DEBUG=false
      - THREADS=8
      - CONTEXT_SIZE=8192
      - MODELS_PATH=/build/models
      - GALLERIES=[{"name":"localai","url":"github:mudler/LocalAI/gallery/index.yaml@master"}]
      - LOCALAI_API_KEY=${LOCALAI_API_KEY}
    volumes:
      - ./models:/build/models
      - ./images:/tmp/generated/images
      - ./backend-data:/build/backend-data
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Create a .env file with an API key (we will use this later for auth):

cat > .env <<EOF
LOCALAI_API_KEY=$(openssl rand -hex 32)
EOF
chmod 600 .env

Start the stack:

docker compose up -d

Watch the first-run logs — the AIO image downloads its default models on first boot (can take 5–10 minutes depending on bandwidth):

docker compose logs -f localai

When you see [GIN-debug] Listening and serving HTTP on :8080, LocalAI is ready.

Quick health check:

curl http://localhost:8080/readyz

Expected output:

OK

Step 6: Browse the Model Gallery and Install Models

LocalAI ships with a curated gallery of 200+ models. You can browse it via the built-in web UI or the REST API.

Web UI

Open http://your-server-ip:8080 in a browser. You land on the LocalAI dashboard with tabs for Chat, Image Generation, TTS, and Models. The Models tab is a clickable gallery — find a model, hit Install, and LocalAI pulls the weights and writes a config automatically.

Install via API

List gallery models:

source .env
curl -H "Authorization: Bearer $LOCALAI_API_KEY" \
  http://localhost:8080/models/available | jq '.[] | .name' | head -30

Install Llama 3.1 8B Instruct (general-purpose chat):

curl -X POST \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id":"llama-3.1-8b-instruct"}' \
  http://localhost:8080/models/apply

The response includes a uuid — poll the job to watch progress:

curl -H "Authorization: Bearer $LOCALAI_API_KEY" \
  http://localhost:8080/models/jobs/<uuid>

Other useful gallery picks:

# Mistral 7B Instruct v0.3 — strong general model, multilingual
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"mistral-7b-instruct-v0.3"}' http://localhost:8080/models/apply
Phi-3.5 Mini — Microsoft, 3.8B, great reasoning per GB
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"phi-3.5-mini-instruct"}' http://localhost:8080/models/apply
Gemma 2 9B — Google DeepMind, balanced quality and speed
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"gemma-2-9b-it"}' http://localhost:8080/models/apply
bge-large-en-v1.5 — top embedding model for RAG
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"bge-large-en-v1.5"}' http://localhost:8080/models/apply
Stable Diffusion XL base 1.0 — image generation
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"stablediffusion-xl-base-1.0"}' http://localhost:8080/models/apply
whisper-large-v3 — speech-to-text
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"whisper-large-v3"}' http://localhost:8080/models/apply
piper-en-amy — English TTS voice
curl -X POST -H "Authorization: Bearer $LOCALAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"id":"voice-en-us-amy-low"}' http://localhost:8080/models/apply

Check what is currently installed:

curl -H "Authorization: Bearer $LOCALAI_API_KEY" \
  http://localhost:8080/v1/models | jq '.data[].id'

Model Size Reference

Step 7: Call the OpenAI-Compatible API

The entire point of LocalAI is that it speaks the OpenAI API. Every example below would also work against api.openai.com by swapping the URL and key.

Chat Completions (curl)

source .env
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instruct",
    "messages": [
      {"role": "system", "content": "You are a concise Linux admin assistant."},
      {"role": "user", "content": "How do I find large files on Ubuntu?"}
    ],
    "temperature": 0.3
  }'

Streaming

Add "stream": true to get server-sent events:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instruct",
    "messages": [{"role":"user","content":"Write a haiku about NVMe disks."}],
    "stream": true
  }'

Python with the Official OpenAI SDK

from openai import OpenAI
client = OpenAI(
    base_url="http://your-server-ip:8080/v1",
    api_key="<contents of LOCALAI_API_KEY>"
)
resp = client.chat.completions.create(
    model="llama-3.1-8b-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful DevOps assistant."},
        {"role": "user", "content": "Explain what a systemd unit override is."}
    ]
)
print(resp.choices[0].message.content)

Note: no code changes from an OpenAI deployment — only base_url and api_key.

Function Calling

LocalAI supports OpenAI-style tool use for models that were trained for it (Llama 3.1, Mistral, Hermes):

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instruct",
    "messages": [{"role":"user","content":"What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'

Step 8: Generate Embeddings

Embeddings power RAG, semantic search, and clustering. LocalAI exposes them at /v1/embeddings — the same endpoint OpenAI uses.

curl http://localhost:8080/v1/embeddings \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-large-en-v1.5",
    "input": "VPS-Server.host offers white-label cloud hosting across Europe."
  }'

Response returns a 1024-dimensional vector that you can store in pgvector, Qdrant, Weaviate, or Chroma. The bge-large-en-v1.5 model consistently ranks near the top of the MTEB leaderboard and runs in ~2 GB of RAM — an easy win over the OpenAI text-embedding-3-small default for English-only workloads.

For multilingual or code-heavy corpora, try bge-m3 or nomic-embed-text-v1.5 from the gallery.

Step 9: Image Generation with Stable Diffusion

If you installed stablediffusion-xl-base-1.0 in Step 6, you can generate images via the /v1/images/generations endpoint:

curl http://localhost:8080/v1/images/generations \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stablediffusion-xl-base-1.0",
    "prompt": "a photorealistic data center aisle at golden hour, cinematic lighting, 35mm lens",
    "size": "1024x1024",
    "n": 1
  }'

The response includes a URL served by LocalAI (from the ./images volume you mounted). Files land in /opt/localai/images/ on the host so you can serve them directly via Nginx or upload them to object storage.

Note: SDXL on CPU is slow — expect 90–180 seconds per image on an 8 vCPU server. On a single NVIDIA A40 or L40S, the same image renders in under 5 seconds. This is the main reason we recommend the GPU variant for image generation at any production volume.

For image editing (inpainting) and img2img, LocalAI also exposes /v1/images/edits and passes the optional init_image / mask_image parameters through to the Diffusers backend.

Step 10: Text-to-Speech and Speech-to-Text

Text-to-Speech

With a Piper voice installed (voice-en-us-amy-low), synthesize speech:

curl http://localhost:8080/v1/audio/speech \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "voice-en-us-amy-low",
    "input": "Your VPS is provisioned and ready to serve traffic.",
    "voice": "amy"
  }' \
  --output speech.wav

Piper voices are tiny (~60 MB each) and run in real-time on CPU. For higher-quality prosody and voice cloning, install Bark (bark-cpp) from the gallery — trades ~2 GB of RAM and slower synthesis for dramatically better output.

Speech-to-Text

With whisper-large-v3 installed, transcribe audio:

curl http://localhost:8080/v1/audio/transcriptions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -F "model=whisper-large-v3" \
  -F "[email protected]"

Response is an OpenAI-shaped JSON with a text field containing the transcription. Whisper-large-v3 handles 99+ languages and detects the spoken language automatically.

Step 11: Secure the API with Nginx + SSL

Exposing LocalAI directly on port 8080 without TLS is fine on a private network but unacceptable on the public internet. Front it with Nginx and Let's Encrypt.

Install Nginx and Certbot:

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/localai:

server {
    listen 80;
    server_name ai.yourdomain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name ai.yourdomain.com;
    ssl_certificate     /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem;
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options DENY;
    # Allow large multipart uploads for audio / images
    client_max_body_size 100m;
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Streaming responses
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}

Enable it and issue a certificate:

sudo ln -s /etc/nginx/sites-available/localai /etc/nginx/sites-enabled/
sudo nginx -t
sudo certbot --nginx -d ai.yourdomain.com
sudo systemctl reload nginx

Also bind LocalAI to localhost only by editing docker-compose.yml:

ports:
      - "127.0.0.1:8080:8080"

Then docker compose up -d. The public surface is now https://ai.yourdomain.com, TLS-terminated by Nginx and gated by the LOCALAI_API_KEY bearer token you set in .env.

Add firewall rules:

sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 443
sudo ufw enable

Performance Tuning

Threading

Set THREADS to match physical CPU cores (not hyperthreaded cores). On an 8 vCPU VPS, THREADS=6 typically outperforms THREADS=8 because leaving 1–2 cores free for I/O and the container runtime reduces contention.

Context Size

Larger CONTEXT_SIZE values allocate more KV cache memory per request. For a 7B model at Q4:

• 2048 tokens: ~4.5 GB RAM per loaded model
• 4096 tokens: ~5.5 GB
• 8192 tokens: ~7.5 GB
• 16384 tokens: ~11 GB

Model Unloading

LOCALAI_WATCHDOG_IDLE=true with LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m unloads models after 15 minutes of inactivity — critical on memory-constrained servers running multiple models.

GPU Layer Offload (GPU Builds)

LocalAI auto-detects GPU memory and offloads as many layers as possible. To force a specific number, add f16: true and gpu_layers: 35 to the model YAML in ./models/<model>.yaml.

Concurrency

LocalAI serializes requests per model by default. For higher throughput on GPU, enable parallel requests by setting parallel_requests: true in the model YAML — useful when paired with vLLM-backed models.

Troubleshooting

Viewing Logs

docker compose logs -f localai           # follow live
docker compose logs --tail 200 localai   # last 200 lines

FAQ

Does LocalAI really implement the full OpenAI API?

Yes — chat completions (including streaming, function calling, and tool use), text completions, embeddings, image generation, image edits, audio transcription, audio translation, text-to-speech, and models listing all work with unmodified OpenAI SDKs. A handful of newer OpenAI-specific features like the Assistants API and Realtime API are either partial or experimental; check the LocalAI docs for the current matrix. Anything built on standard chat/embed/image endpoints — LangChain, LlamaIndex, AutoGen, CrewAI, LibreChat, AnythingLLM — will work by swapping the base URL.

Do I need a GPU to run LocalAI?

No. The CPU image runs every modality — LLM chat, embeddings, TTS, STT, and even Stable Diffusion — entirely on CPU. Chat and embeddings are perfectly usable on a modern 8 vCPU VPS (~15–25 tokens/sec for a 7B model). Image generation on CPU is the weak point: expect 90–180 seconds per SDXL image. If your workload is mostly LLM and embeddings, CPU is fine. If image generation volume is above a handful per hour, a GPU pays for itself quickly.

How does LocalAI compare to Ollama?

Ollama is simpler and lighter — it does chat and embeddings very well and nothing else. LocalAI is broader — it replaces OpenAI, Stability AI, ElevenLabs, and Whisper with one server. If you only need LLM chat, start with Ollama (and optionally put Open WebUI in front). If you need image generation, TTS, or STT under the same auth layer as your LLM, use LocalAI. Many teams run both — Ollama for hot-swapping chat models during development, LocalAI for the production API surface.

How does LocalAI compare to vLLM?

vLLM is purpose-built for maximum GPU throughput on LLM chat — continuous batching, PagedAttention, tensor parallelism across multiple GPUs. It is the right choice when you have dedicated A100 / H100 hardware and need to serve hundreds of concurrent users. It does not do image generation, TTS, or STT. LocalAI can actually use vLLM as a backend for specific models while handling all other modalities through its native backends, giving you the best of both.

Can I run LocalAI behind the same domain as my main app?

Yes. The Nginx config in Step 11 is a path-based reverse proxy and composes cleanly with other server blocks. A common pattern is to expose LocalAI at ai.yourdomain.com (as shown) and your main app at app.yourdomain.com, both on the same VPS, both TLS-terminated by the same Nginx instance. Alternatively you can expose LocalAI at yourdomain.com/ai/ using a location /ai/ { rewrite ^/ai/(.*)$ /$1 break; proxy_pass http://127.0.0.1:8080; } block.

How do I add a custom model that is not in the gallery?

Drop a GGUF file into ./models/ and create a matching YAML config. Minimum example for ./models/my-model.yaml:

name: my-model
backend: llama-cpp
parameters:
  model: my-model.gguf
context_size: 8192
template:
  chat: |
    {{.Input}}

Restart LocalAI (docker compose restart localai) and the model appears at /v1/models. The full schema supports prompt templates, stop tokens, LoRA adapters, and backend-specific tuning — see the LocalAI docs for advanced options.

Does LocalAI support function calling and structured output?

Yes, when the underlying model supports it. Llama 3.1, Mistral, Hermes-2-Pro, and Qwen 2.5 all handle OpenAI-style tools and tool_choice parameters cleanly. For strict JSON schema enforcement, set response_format: {"type": "json_object"} — LocalAI uses grammar-constrained sampling via llama.cpp to guarantee valid JSON output, which is actually more reliable than OpenAI's equivalent feature on smaller models.

Next Steps

Now that LocalAI is running, here are the natural follow-ons:

• Put a polished chat UI in front of it — Install LibreChat and point it at your LocalAI endpoint to get a multi-user, multi-model chat interface with conversation history, prompt templates, and plugin support.

• Build a document Q&A system — Deploy AnythingLLM and connect it to LocalAI for RAG over PDFs, websites, Notion, and Confluence. AnythingLLM uses your LocalAI embeddings model for indexing and your LocalAI chat model for answering.

• Add Open WebUI for a second audience — Open WebUI works with both LocalAI and Ollama simultaneously — useful if part of your team prefers the ChatGPT-style UI while developers use the API directly.

• Scale out LLM inference with vLLM — Once you outgrow single-server LocalAI for chat workloads, front the same API surface with vLLM on GPU and keep LocalAI for the non-LLM modalities (images, TTS, STT).

• Compare to a pure-LLM install — Run our Ollama on Ubuntu guide on a second VPS and benchmark both for your specific workload. Many teams settle on LocalAI for the API and Ollama for the CLI.

• Source references — Official project: localai.io. Source, issues, and release notes: github.com/mudler/LocalAI.

Get LocalAI Running on Tuned Hardware

Our CloudCore Business VPS ships with NVMe storage, 24 GB RAM, and Docker pre-installed — the minimum comfortable footprint for the full LocalAI stack (LLM + embeddings + image + speech). EU-hosted, flat-rate, unlimited bandwidth.

- 8 vCPU / 24 GB RAM / 200 GB NVMe

- Optional NVIDIA GPU add-on (A30 / A40 / L40S)

- Ubuntu 24.04 LTS with Docker and NVIDIA Container Toolkit pre-configured on GPU plans

- One flat price — no per-token billing, ever

Deploy CloudCore Business — from EUR 29.99/month.