How to Install Ollama on Ubuntu 24.04 — Run LLMs Locally on Your VPS

Running large language models on your own infrastructure gives you complete control over your data, eliminates per-token API costs, and removes rate limits entirely. This guide walks you through installing Ollama on an Ubuntu 24.04 VPS, from first SSH connection to a production-hardened deployment serving models like Llama 3.1, Gemma 2, and Mistral.

Skip the setup? Deploy Ollama in one click with our pre-configured AI Server image. Launch an AI-ready VPS now and start prompting in under 60 seconds.

Table of Contents

What is Ollama?

Ollama is an open-source tool that makes it straightforward to download, run, and manage large language models locally. It wraps the complexity of model quantization, memory management, and inference optimization into a single binary with a clean CLI and REST API. Think of it as Docker for LLMs: you pull a model, run it, and start generating text immediately.

Ollama supports a broad and growing library of models. On the general-purpose side, you can run Llama 3.1 (Meta's flagship open model, available in 8B, 70B, and 405B parameter variants), Gemma 2 (Google DeepMind's efficient family in 2B, 9B, and 27B sizes), Mistral and Mixtral (Mistral AI's dense and mixture-of-experts models), and Phi-3 (Microsoft's compact but capable small model). For code-specific workloads, Code Llama and DeepSeek Coder V2 handle code generation, explanation, and refactoring. Specialized models like LLaVA support multimodal inputs (images + text), while nomic-embed-text produces embeddings for retrieval-augmented generation pipelines. The full model library at ollama.com/library lists over 100 models, and you can import any GGUF-format model from Hugging Face.

The use cases for self-hosted LLMs are practical and growing. Development teams use Ollama for AI-powered code completion in editors like VS Code and Neovim via Continue.dev. Companies build private chatbots that never send data to external APIs. Data teams construct RAG (Retrieval-Augmented Generation) pipelines for document analysis and question-answering over internal knowledge bases. DevOps teams use it for log analysis and anomaly detection. And researchers fine-tune models on proprietary datasets without uploading anything to a third-party platform.

Why Self-Host LLMs on Your VPS?

Running models on your own VPS instead of calling cloud APIs offers concrete advantages:

• Complete data privacy -- Your prompts, documents, and generated outputs never leave your server. No data is transmitted to OpenAI, Google, or any third party. This matters for legal documents, medical records, financial data, and proprietary code.
• Predictable, flat-rate cost -- A VPS costs the same whether you send 10 prompts or 10,000. There are no per-token charges, no surprise bills, and no metering.
• No rate limits -- Cloud APIs throttle you during peak demand. Your own server processes requests as fast as the hardware allows, 24/7.
• Full customization -- Choose any model, any quantization level, any system prompt. Fine-tune on your own data. Chain models together. No vendor lock-in.
• GDPR and regulatory compliance -- For businesses handling EU citizen data, keeping AI inference on-premises (or on a known EU-hosted server) simplifies compliance. You control the data lifecycle end to end.
• Offline capability -- Once a model is pulled, it runs without any internet connection. Ideal for air-gapped environments or unreliable connectivity.
• Low latency -- Local inference eliminates network round-trips. When your application and your model share the same machine, response times drop dramatically.

Cost Comparison: Self-Hosted vs. Cloud AI APIs

Prerequisites

Before you begin, make sure you have:

• A VPS running Ubuntu 24.04 LTS with root or sudo access
• SSH access to your server (PuTTY on Windows, or the built-in terminal on macOS/Linux)
• At least 8 GB of RAM for running 7B-parameter models (12 GB+ recommended)
• At least 20 GB of free disk space for Ollama and one model (50 GB+ recommended if you plan to keep multiple models)

Recommended Plan: CloudCore Professional

For running Llama 3.1 8B, Gemma 2 9B, and similar 7B-9B models comfortably, we recommend the CloudCore Professional plan:

- 6 vCPU cores

- 12 GB RAM

- 100 GB NVMe SSD

- Unmetered bandwidth

- EUR 19.99/month

This gives you enough headroom for a model loaded in memory plus your application stack. For larger models (70B+) or GPU-accelerated inference, see our GPU VPS plans.

Connect to your server via SSH to get started:

ssh root@your-server-ip

Step 1: Update System Packages

Start by updating your package index and upgrading installed packages. This ensures you have the latest security patches and that dependency resolution works correctly during the Ollama install.

sudo apt update && sudo apt upgrade -y

Expected output (abbreviated):

Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Reading package lists... Done
Building dependency tree... Done
Calculating upgrade... Done
The following packages will be upgraded:
  ...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

If your kernel was updated, reboot before continuing:

sudo reboot

Then reconnect via SSH after a minute.

Step 2: Install Ollama

Ollama provides an official install script that handles everything: downloading the correct binary for your architecture (amd64 or arm64), creating a dedicated ollama system user, setting up a systemd service, and detecting any NVIDIA or AMD GPUs.

Run the installer:

curl -fsSL https://ollama.com/install.sh | sh

Expected output:

>>> Installing ollama to /usr/local/bin...
>>> Creating ollama user...
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.

The script performs these actions:

The service is configured to start automatically on boot, so Ollama will survive reboots without any additional setup.

Step 3: Verify the Installation

Confirm that Ollama is installed and the service is running.

Check the installed version:

ollama --version

Expected output:

ollama version is 0.6.2

Check that the systemd service is active:

sudo systemctl status ollama

Expected output:

● ollama.service - Ollama Service
     Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-04-16 10:00:00 UTC; 1min ago
   Main PID: 1234 (ollama)
      Tasks: 9 (limit: 14236)
     Memory: 42.0M
        CPU: 1.234s
     CGroup: /system.slice/ollama.service
             └─1234 /usr/local/bin/ollama serve

Verify the API endpoint is responding:

curl http://localhost:11434

Expected output:

Ollama is running

List currently available models (will be empty at this point):

ollama list

Expected output:

NAME    ID    SIZE    MODIFIED

Everything is working. Time to download a model.

Step 4: Pull Your First Model

Ollama downloads models from its registry with the pull command. We recommend starting with Llama 3.1 8B, which offers an excellent balance of quality and resource usage.

ollama pull llama3.1

Expected output:

pulling manifest
pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB
pulling 4fa551d4f938... 100% ▕████████████████▏  12 KB
pulling 8ab4849b038c... 100% ▕████████████████▏   254 B
pulling 577073ffcc6c... 100% ▕████████████████▏   110 B
pulling ad1518640c43... 100% ▕████████████████▏   483 B
verifying sha256 digest
writing manifest
success

The download size for the default (Q4_0 quantized) Llama 3.1 8B is approximately 4.7 GB. On a VPS with good bandwidth, this typically takes 1-3 minutes.

Model Sizes and RAM Requirements

Choosing the right model depends on your VPS resources. Here is a reference table for popular models:

Step 5: Run and Chat with the Model

Start an interactive chat session with the model you just downloaded:

ollama run llama3.1

You will see a prompt where you can type messages directly:

>>> Send a message (/? for help)

Try a few prompts:

>>> What are the main advantages of using a VPS over shared hosting?

The model will stream its response token by token. A typical reply on a 6 vCPU server takes 5-15 seconds for the first token (loading), then generates at approximately 10-20 tokens per second on CPU.

You can also set a system prompt at launch:

ollama run llama3.1 "You are a Linux sysadmin expert. Be concise."

Useful commands inside the interactive session:

• /bye -- Exit the chat
• /show info -- Display model details (parameters, quantization, context length)
• /set parameter temperature 0.3 -- Adjust generation temperature
• /set parameter num_ctx 4096 -- Set the context window size
• /clear -- Clear the conversation history

>>> /bye

Step 6: Use the REST API

Ollama exposes a REST API on port 11434 that lets any application send prompts and receive responses. This is how you integrate Ollama into web apps, scripts, and automation pipelines.

Generate a Completion (Single Prompt)

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Explain what a reverse proxy is in two sentences.",
  "stream": false
}'

Expected output (JSON):

{
  "model": "llama3.1",
  "created_at": "2026-04-16T10:30:00.000000Z",
  "response": "A reverse proxy is a server that sits between client devices and backend servers, forwarding client requests to the appropriate backend and returning the responses. It is commonly used for load balancing, SSL termination, caching, and protecting backend servers from direct exposure to the internet.",
  "done": true,
  "done_reason": "stop",
  "context": [...]
}

Setting "stream": false returns the complete response in a single JSON object. If you omit it or set "stream": true, Ollama streams newline-delimited JSON objects as each token is generated -- useful for real-time UIs.

Streaming Example

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "List three uses for a VPS."
}'

Each line of output is a JSON object containing one or more tokens:

{"model":"llama3.1","created_at":"2026-04-16T10:30:01Z","response":"1","done":false}
{"model":"llama3.1","created_at":"2026-04-16T10:30:01Z","response":".","done":false}
{"model":"llama3.1","created_at":"2026-04-16T10:30:01Z","response":" Web","done":false}
...
{"model":"llama3.1","created_at":"2026-04-16T10:30:05Z","response":"","done":true,"done_reason":"stop"}

Chat Endpoint (Multi-Turn Conversation)

The /api/chat endpoint supports multi-turn conversations with message roles:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful server administration assistant."
    },
    {
      "role": "user",
      "content": "How do I check disk usage on Ubuntu?"
    }
  ],
  "stream": false
}'

Expected output:

{
  "model": "llama3.1",
  "created_at": "2026-04-16T10:31:00.000000Z",
  "message": {
    "role": "assistant",
    "content": "You can check disk usage on Ubuntu using the df command for filesystem-level usage and du for directory-level usage:\n\n

• OLLAMA_HOST=0.0.0.0:11434 -- By default, Ollama only listens on 127.0.0.1. Setting this to 0.0.0.0 allows external connections (required if other servers or a reverse proxy on a different machine need access). If you only need local access, keep the default 127.0.0.1.
• OLLAMA_MODELS=/var/lib/ollama/models -- The directory where model files are stored. Change this if you want to store models on a larger mounted volume.
• OLLAMA_NUM_PARALLEL=2 -- Number of requests Ollama will process concurrently for the same model. Set this based on your RAM: each parallel slot requires additional memory for the context window.
• OLLAMA_MAX_LOADED_MODELS=2 -- Maximum number of models to keep loaded in memory simultaneously. Each loaded model consumes RAM even when idle. Set to 1 on memory-constrained servers.
• OLLAMA_KEEP_ALIVE=10m -- How long to keep an idle model loaded in memory. Default is 5m. Set to 0 to unload immediately after each request (saves RAM, adds latency on next request), or -1 to keep models loaded indefinitely.

If you have mounted additional storage (for example, a block storage volume at /mnt/data), move the model directory:</code></pre></div>bash sudo systemctl stop ollama sudo mv /var/lib/ollama/models /mnt/data/ollama-models sudo ln -s /mnt/data/ollama-models /var/lib/ollama/models sudo systemctl start ollama <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Or update the OLLAMA_MODELS environment variable in the override file to point to the new location directly.

Step 8: Set Up GPU Acceleration (Optional)

Running models on a GPU dramatically increases inference speed -- typically 5-10x faster than CPU-only. If your VPS has an NVIDIA GPU, Ollama will detect and use it automatically once the drivers are installed.

Install NVIDIA Drivers</code></pre></div>bash

By default, Ollama listens only on 127.0.0.1:11434, which is safe -- no external access is possible. If you changed OLLAMA_HOST to 0.0.0.0 in Step 7, you need to secure it.

Option A: Firewall Rules with UFW (Simplest)

If only your application on the same server needs the API, keep OLLAMA_HOST=127.0.0.1 and ensure port 11434 is not open:</code></pre></div>bash sudo ufw deny 11434 sudo ufw enable sudo ufw status <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">### Option B: Nginx Reverse Proxy with SSL and Basic Auth

For external access (such as connecting from a separate frontend server), set up Nginx as a reverse proxy with TLS and authentication.

Install Nginx and the Apache utilities package (for htpasswd):</code></pre></div>bash sudo apt install -y nginx apache2-utils <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Create a password file:</code></pre></div>bash sudo htpasswd -c /etc/nginx/.ollama-htpasswd apiuser <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">You will be prompted to enter and confirm a password.

Create the Nginx configuration:</code></pre></div>bash sudo tee /etc/nginx/sites-available/ollama > /dev/null <<'EOF' server { listen 443 ssl; server_name ollama.yourdomain.com;

ssl_certificate /etc/letsencrypt/live/ollama.yourdomain.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ollama.yourdomain.com/privkey.pem;

# Security headers add_header X-Content-Type-Options nosniff; add_header X-Frame-Options DENY;

# Limit request body size (adjust for your largest expected prompt) client_max_body_size 10m;

location / { auth_basic "Ollama API"; auth_basic_user_file /etc/nginx/.ollama-htpasswd;

proxy_pass http://127.0.0.1:11434; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme;

# Required for streaming responses proxy_buffering off; proxy_read_timeout 600s; proxy_send_timeout 600s; } }

server { listen 80; server_name ollama.yourdomain.com; return 301 https://$host$request_uri; } EOF <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Enable the site and obtain an SSL certificate with Certbot:</code></pre></div>bash sudo ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/ sudo apt install -y certbot python3-certbot-nginx sudo certbot --nginx -d ollama.yourdomain.com sudo nginx -t && sudo systemctl reload nginx <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Now you can access the API externally with authentication:</code></pre></div>bash curl -u apiuser:yourpassword https://ollama.yourdomain.com/api/generate -d '{ "model": "llama3.1", "prompt": "Hello!", "stream": false }' <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">### Option C: Restrict by IP Address

If only specific IP addresses need access, use UFW to allow only those:</code></pre></div>bash sudo ufw allow from 203.0.113.50 to any port 11434 sudo ufw deny 11434 <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">---

Post-Install: Run Multiple Models

One of Ollama's strengths is managing multiple models side by side. Pull additional models for different use cases:</code></pre></div>bash

Google's efficient general-purpose model

Limit how many models stay loaded in RAM simultaneously:</code></pre></div>bash Environment="OLLAMA_MAX_LOADED_MODELS=1" <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Set this to 1 on servers with 12 GB RAM. Each loaded 7B model consumes approximately 6-8 GB. Loading a second model while one is active can trigger OOM (out-of-memory) kills.

Context Window Size

The context window determines how much conversation history the model can reference. Larger windows use more memory.

Set it per-request via the API:</code></pre></div>bash curl http://localhost:11434/api/generate -d '{ "model": "llama3.1", "prompt": "Summarize this document...", "options": { "num_ctx": 8192 }, "stream": false }' <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Or set it in the interactive CLI:</code></pre></div> >>> /set parameter num_ctx 8192 <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">Context window memory cost scales linearly. At Q4_0 quantization for a 7B model:

• 2048 tokens: ~5.5 GB RAM
• 4096 tokens: ~6.5 GB RAM (default)
• 8192 tokens: ~8.5 GB RAM
• 16384 tokens: ~12.5 GB RAM

The most useful debugging tool is the Ollama service log:</code></pre></div>bash sudo journalctl -u ollama -f <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">This streams logs in real time. Press Ctrl+C to stop. To see the last 50 lines:</code></pre></div>bash sudo journalctl -u ollama -n 50 --no-pager <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text">---

FAQ

What are the minimum hardware requirements to run Ollama?

Ollama itself is lightweight -- the binary is under 100 MB. The hardware requirements depend entirely on the model you want to run. The smallest useful models (Phi-3 Mini at 3.8B parameters, Gemma 2 at 2B) run on servers with as little as 4 GB of RAM. For the popular 7B-9B parameter models like Llama 3.1 and Mistral, plan for 8-12 GB of RAM. Storage needs vary from 2 GB to 40+ GB per model depending on size and quantization. CPU-wise, more cores help with parallel requests, but even a 2 vCPU server can run a single 7B model for personal use.

Is a GPU required to run Ollama?

No. Ollama runs entirely on CPU and works well for many use cases without a GPU. On a 6 vCPU server, expect approximately 10-20 tokens per second with a 7B model, which is perfectly usable for chatbots, APIs, and batch processing. A GPU accelerates inference significantly (50-100+ tokens per second on an NVIDIA A30 or A100), making it worthwhile for high-throughput production workloads or larger models. Ollama auto-detects GPUs -- you do not need to configure anything beyond installing the drivers.

Can I run multiple models at the same time?

Yes. Ollama can keep multiple models loaded in memory simultaneously, controlled by the OLLAMA_MAX_LOADED_MODELS environment variable. Each model consumes RAM independently. For example, running both Llama 3.1 8B (~8 GB) and Mistral 7B (~7 GB) simultaneously requires at least 17 GB of available RAM plus overhead. On memory-constrained servers, set OLLAMA_MAX_LOADED_MODELS=1 so Ollama unloads the current model before loading a new one. The unload/reload cycle adds a few seconds of latency but prevents out-of-memory conditions.

How does Ollama compare to vLLM and LocalAI?

Ollama focuses on simplicity and developer experience. It is the easiest to install, has a clean CLI, bundles model management, and works on CPU or GPU with no configuration. Best for: individual developers, small teams, prototyping, and single-server deployments.

vLLM is a high-performance inference engine optimized for GPU throughput. It supports PagedAttention, continuous batching, and tensor parallelism across multiple GPUs. Best for: production deployments at scale where you need maximum tokens per second and are running GPU clusters.

LocalAI aims to be a drop-in OpenAI API replacement, supporting not just text generation but also image generation, audio transcription, and embeddings with the same API schema. Best for: projects that need OpenAI-compatible endpoints with multiple modalities.

For most VPS users deploying 1-3 models on a single server, Ollama is the right choice. Move to vLLM when you need multi-GPU scaling or continuous batching for dozens of concurrent users.

How do I integrate Ollama with LangChain or other frameworks?

Ollama's REST API is compatible with most LLM frameworks out of the box. For LangChain (Python):</code></pre></div>bash pip install langchain-ollama <div class="code-block" data-lang="text"><div class="code-block__header"><span class="code-block__lang">text</span></div><pre><code class="language-text"></code></pre></div>python from langchain_ollama import OllamaLLM

llm = OllamaLLM(model="llama3.1", base_url="http://localhost:11434") response = llm.invoke("What is a VPS?") `

For LlamaIndex, use the llama-index-llms-ollama package. For JavaScript/TypeScript, the official ollama npm package provides a typed client. Ollama also supports the OpenAI-compatible endpoint at /v1/chat/completions, so any tool that works with the OpenAI API can point to your Ollama server with a one-line URL change.

Next Steps

Now that Ollama is running on your VPS, here are recommended next steps to build on your setup:

• Install Open WebUI for a ChatGPT-like interface -- Open WebUI gives you a polished browser-based chat interface that connects to your Ollama instance. Deploy it with a single Docker command and share access with your team.

• Explore the model library -- Browse the full catalog at ollama.com/library. Try specialized models like sqlcoder for SQL generation, medllama2 for medical text, or llava` for image understanding.

Skip the Manual Install -- Get Ollama Pre-Installed

Our AI-Ready VPS plans come with Ollama, Open WebUI, and your choice of models pre-configured. Deploy in 60 seconds and start building immediately.

- Llama 3.1 8B pre-loaded and ready to prompt

- Open WebUI accessible at your domain

- Nginx reverse proxy with SSL configured

- systemd service with production-tuned settings

- Optional GPU acceleration on GPU VPS plans

Deploy Your AI VPS Now -- Plans start at EUR 19.99/month.