How to Install Google Gemma 4 on Ubuntu 24.04 — Self-Hosted Open LLM

Quick Summary

Google Gemma 4 is Google's latest family of open-weights large language models, bringing Gemini-class reasoning, multilingual fluency, and strong code generation to a model you can run entirely on your own infrastructure. Whether you need privacy-first AI for processing sensitive documents, a local coding assistant, or an inference engine for your SaaS product, Gemma 4 delivers flagship-tier quality without sending a single byte to an external API.

In this guide, you will learn three methods to install and run Gemma 4 on Ubuntu 24.04: Ollama (the fastest path), Hugging Face Transformers (maximum flexibility), and vLLM (production-grade throughput). Every command is tested on real VPS hardware, and every Python script is complete and runnable.

Just want it running in 5 minutes? Jump to Method 1: Install via Ollama. Need a VPS that can handle it? Our CloudCore Business plan gives you 8 vCPU, 24 GB RAM, and 200 GB NVMe starting at EUR 35.99/mo, which comfortably runs the 12B parameter model with CPU inference.

Table of Contents

What is Google Gemma 4?

Gemma 4 is the fourth generation of Google's open-weights language model family. Built by the same Google DeepMind research teams behind Gemini, Gemma 4 distills large-scale model capabilities into sizes that fit on a single server or even a modest VPS. Unlike proprietary models locked behind API paywalls, Gemma 4 ships with open weights under a permissive license that allows commercial use, fine-tuning, and redistribution.

Key Capabilities

• Multilingual understanding: Fluent in over 30 languages with strong performance on non-English tasks including translation, summarization, and question answering.
• Code generation: Trained on a large code corpus covering Python, JavaScript, TypeScript, Go, Rust, Java, C++, and more. Suitable for code completion, refactoring, and explanation tasks.
• Reasoning and math: Improved chain-of-thought reasoning compared to previous Gemma generations, with measurable gains on GSM8K, MATH, and ARC benchmarks.
• Instruction following: The instruction-tuned (IT) variants respond accurately to structured prompts, system messages, and multi-turn conversations.
• Long context: Supports context windows up to 128K tokens on the larger variants, enabling document analysis, codebase understanding, and multi-document summarization.

How Gemma 4 Compares

Why Self-Host Gemma 4?

Running Gemma 4 on your own VPS instead of calling the Gemini API gives you concrete, measurable advantages across cost, privacy, performance, and control.

Privacy and Data Sovereignty

When you self-host, every prompt and every response stays on your server. No data is transmitted to Google, no prompts are logged by a third party, and no content is used for model training. This matters for healthcare, legal, financial, and government workloads where data residency rules apply.

Cost at Scale

API pricing adds up fast once you move past prototyping. Here is what the numbers look like for a moderate workload of 10 million input tokens and 2 million output tokens per month:

No Rate Limits

The Gemini API enforces requests-per-minute and tokens-per-minute limits. Self-hosted Gemma 4 processes requests as fast as your hardware allows, with no artificial throttling. This is critical for batch processing, RAG pipelines, and real-time applications.

Customization and Fine-Tuning

With full access to the model weights, you can fine-tune Gemma 4 on your domain-specific data using LoRA or QLoRA. This means you can train the model on your company's documentation, coding style, product catalog, or industry terminology -- something the Gemini API does not offer.

Latency

Local inference eliminates network round-trips. On a GPU-equipped VPS, Gemma 4 responds in milliseconds rather than the 200-500ms minimum you see with API calls. For chatbots and real-time applications, this difference is noticeable.

Offline and Air-Gapped Use

Once downloaded, Gemma 4 runs without any internet connection. This enables deployment in air-gapped environments, on-premise data centers, and edge devices where connectivity is intermittent or prohibited.

Gemma 4 Model Variants

Gemma 4 ships in multiple sizes. Smaller models run on CPU-only VPS hardware; larger models need dedicated GPUs. Here is the full lineup with hardware requirements:

Prerequisites

Before you begin, make sure you have the following:

• A VPS running Ubuntu 24.04 LTS. We recommend the CloudCore Business plan (8 vCPU, 24 GB RAM, 200 GB NVMe SSD) for the 12B model. For the 27B model, you need a GPU-equipped server.
• Root or sudo access via SSH.
• At least 50 GB of free disk space for model weights and dependencies. The 12B quantized model is about 7 GB, but you need extra room for the runtime, swap, and temporary files during download.
• A stable internet connection for the initial model download. After that, no internet is required.

ssh root@your-server-ip
lsb_release -a

You should see Ubuntu 24.04 in the output. Update your system before proceeding:

sudo apt update && sudo apt upgrade -y

Method 1: Install via Ollama (Recommended)

Ollama is the fastest way to get Gemma 4 running. It handles model downloading, quantization, memory management, and API serving in a single binary. No Python environment, no dependency conflicts, no manual configuration.

Step 1: Install Ollama

Run the official installer:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation:

ollama --version

You should see version 0.6 or later. Ollama installs as a systemd service and starts automatically.

Step 2: Pull the Gemma 4 Model

Download the 12B instruction-tuned model (quantized, about 7 GB):

ollama pull gemma4:12b

For smaller hardware, pull the 4B variant:

ollama pull gemma4:4b

For the largest model with highest quality:

ollama pull gemma4:27b

The download takes a few minutes depending on your server's bandwidth. Ollama stores models in /usr/share/ollama/.ollama/models/ by default.

Step 3: Run an Interactive Chat

Start a conversation directly from your terminal:

ollama run gemma4:12b

You will see a prompt where you can type messages:

>>> What is the capital of France?
The capital of France is Paris. It is the largest city in France and serves
as the country's political, economic, and cultural center.
>>> Write a Python function to calculate fibonacci numbers.
Here's a Python function to calculate Fibonacci numbers:
def fibonacci(n):
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        a, b = 0, 1
        for _ in range(2, n + 1):
            a, b = b, a + b
        return b
>>> /bye

Press Ctrl+D or type /bye to exit.

Step 4: Use the REST API

Ollama exposes an OpenAI-compatible API on port 11434. You can call it from any application, language, or framework.

Generate a completion:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4:12b",
  "prompt": "Explain quantum computing in simple terms.",
  "stream": false
}'

Chat with message history (OpenAI-compatible):

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "gemma4:12b",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are the benefits of containerization?"}
  ]
}'

List available models:

curl http://localhost:11434/api/tags

The API returns JSON responses. The /v1/chat/completions endpoint is drop-in compatible with the OpenAI SDK, so any application that works with OpenAI can point to your Ollama server instead.

Step 5: Configure for Production

By default, Ollama only listens on localhost. For production use, you need to configure it to accept remote connections, set resource limits, and ensure it starts on boot.

Edit the systemd service to set environment variables:

sudo systemctl edit ollama

Add the following in the override file that opens:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_KEEP_ALIVE=10m"

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify the service is running:

sudo systemctl status ollama

Explanation of environment variables:

• OLLAMA_HOST=0.0.0.0:11434 -- Listen on all interfaces (required if you want remote access; secure with a reverse proxy as shown in the Security section).
• OLLAMA_NUM_PARALLEL=4 -- Handle up to 4 concurrent requests.
• OLLAMA_MAX_LOADED_MODELS=2 -- Keep up to 2 models loaded in memory simultaneously.
• OLLAMA_KEEP_ALIVE=10m -- Unload models from memory after 10 minutes of inactivity to free RAM.

Method 2: Install via Hugging Face Transformers

The Hugging Face Transformers library gives you full control over model loading, inference parameters, tokenization, and pipeline construction. This method is ideal when you need to integrate Gemma 4 into a custom Python application, chain it with other models, or build a tailored API.

Step 1: Install Python 3.11+ and pip

Ubuntu 24.04 ships with Python 3.12. Verify it is available:

python3 --version

If you see Python 3.12.x, you are ready. Install pip and the venv module:

sudo apt install -y python3-pip python3-venv python3-dev

Step 2: Create a Virtual Environment

Always use a virtual environment to avoid conflicts with system packages:

mkdir -p ~/gemma4-project
cd ~/gemma4-project
python3 -m venv venv
source venv/bin/activate

Step 3: Install PyTorch and Transformers

Install PyTorch with CUDA support (if you have a GPU) or CPU-only:

With GPU (CUDA 12.x):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install transformers accelerate sentencepiece protobuf huggingface_hub

CPU only:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install transformers accelerate sentencepiece protobuf huggingface_hub

Step 4: Download the Model

Gemma 4 models on Hugging Face require you to accept the license agreement. Visit the model page at https://huggingface.co/google/gemma-4-12b-it, click "Agree and access repository," then authenticate your CLI:

huggingface-cli login

Paste your Hugging Face access token when prompted. Then download the model:

huggingface-cli download google/gemma-4-12b-it --local-dir ./models/gemma-4-12b-it

This downloads approximately 24 GB of model weights in safetensors format. For a smaller footprint, you can download the 4B variant:

huggingface-cli download google/gemma-4-4b-it --local-dir ./models/gemma-4-4b-it

Step 5: Run Inference

Create a file called inference.py with the following complete script:

#!/usr/bin/env python3
"""
Gemma 4 inference script using Hugging Face Transformers.
Runs on GPU if available, falls back to CPU with automatic dtype selection.
"""
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_PATH = "./models/gemma-4-12b-it"
def load_model():
    """Load the Gemma 4 model and tokenizer."""
    print("Loading tokenizer...")
    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
    print("Loading model...")
    if torch.cuda.is_available():
        model = AutoModelForCausalLM.from_pretrained(
            MODEL_PATH,
            device_map="auto",
            torch_dtype=torch.bfloat16,
        )
        print(f"Model loaded on GPU: {torch.cuda.get_device_name(0)}")
    else:
        model = AutoModelForCausalLM.from_pretrained(
            MODEL_PATH,
            device_map="cpu",
            torch_dtype=torch.float32,
        )
        print("Model loaded on CPU (inference will be slower)")
    return model, tokenizer
def generate_response(model, tokenizer, user_message, system_message=None, max_new_tokens=1024):
    """Generate a response from the model."""
    messages = []
    if system_message:
        messages.append({"role": "system", "content": system_message})
    messages.append({"role": "user", "content": user_message})
    input_text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            top_k=50,
            repetition_penalty=1.1,
        )
    response_ids = outputs[0][inputs["input_ids"].shape[1]:]
    response_text = tokenizer.decode(response_ids, skip_special_tokens=True)
    return response_text
def main():
    model, tokenizer = load_model()
    print("\nGemma 4 is ready. Type your message (or 'quit' to exit).\n")
    while True:
        user_input = input("You: ").strip()
        if user_input.lower() in ("quit", "exit", "q"):
            print("Goodbye.")
            break
        if not user_input:
            continue
        print("Gemma 4: ", end="", flush=True)
        response = generate_response(
            model,
            tokenizer,
            user_input,
            system_message="You are a helpful, accurate, and concise assistant.",
        )
        print(response)
        print()
if __name__ == "__main__":
    main()

Run it:

python inference.py

Step 6: Set Up as an API with FastAPI

For production applications, wrap the model in a FastAPI server that exposes an OpenAI-compatible chat endpoint. Create a file called api_server.py:

#!/usr/bin/env python3
"""
FastAPI server for Gemma 4 with OpenAI-compatible chat completions endpoint.
Run with: uvicorn api_server:app --host 0.0.0.0 --port 8000
"""
import time
import uuid
from contextlib import asynccontextmanager
import torch
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_PATH = "./models/gemma-4-12b-it"
model = None
tokenizer = None
@asynccontextmanager
async def lifespan(app: FastAPI):
    """Load model on startup, release on shutdown."""
    global model, tokenizer
    print("Loading Gemma 4 model...")
    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
    if torch.cuda.is_available():
        model = AutoModelForCausalLM.from_pretrained(
            MODEL_PATH,
            device_map="auto",
            torch_dtype=torch.bfloat16,
        )
    else:
        model = AutoModelForCausalLM.from_pretrained(
            MODEL_PATH,
            device_map="cpu",
            torch_dtype=torch.float32,
        )
    print("Model loaded and ready.")
    yield
    del model
    del tokenizer
    torch.cuda.empty_cache()
app = FastAPI(title="Gemma 4 API", lifespan=lifespan)
class Message(BaseModel):
    role: str
    content: str
class ChatCompletionRequest(BaseModel):
    model: str = "gemma-4-12b-it"
    messages: list[Message]
    max_tokens: int = 1024
    temperature: float = 0.7
    top_p: float = 0.9
    stream: bool = False
class ChatCompletionResponse(BaseModel):
    id: str
    object: str = "chat.completion"
    created: int
    model: str
    choices: list[dict]
    usage: dict
@app.post("/v1/chat/completions")
async def chat_completions(request: ChatCompletionRequest):
    """OpenAI-compatible chat completions endpoint."""
    if model is None:
        raise HTTPException(status_code=503, detail="Model not loaded")
    messages = [{"role": m.role, "content": m.content} for m in request.messages]
    input_text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
    input_token_count = inputs["input_ids"].shape[1]
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=request.max_tokens,
            do_sample=True,
            temperature=request.temperature,
            top_p=request.top_p,
            top_k=50,
            repetition_penalty=1.1,
        )
    response_ids = outputs[0][input_token_count:]
    response_text = tokenizer.decode(response_ids, skip_special_tokens=True)
    output_token_count = len(response_ids)
    return ChatCompletionResponse(
        id=f"chatcmpl-{uuid.uuid4().hex[:12]}",
        created=int(time.time()),
        model=request.model,
        choices=[
            {
                "index": 0,
                "message": {"role": "assistant", "content": response_text},
                "finish_reason": "stop",
            }
        ],
        usage={
            "prompt_tokens": input_token_count,
            "completion_tokens": output_token_count,
            "total_tokens": input_token_count + output_token_count,
        },
    )
@app.get("/v1/models")
async def list_models():
    """List available models."""
    return {
        "object": "list",
        "data": [
            {
                "id": "gemma-4-12b-it",
                "object": "model",
                "owned_by": "google",
            }
        ],
    }
@app.get("/health")
async def health():
    return {"status": "ok", "model_loaded": model is not None}

Install FastAPI and Uvicorn, then start the server:

pip install fastapi uvicorn
uvicorn api_server:app --host 0.0.0.0 --port 8000

Test it with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-12b-it",
    "messages": [
      {"role": "user", "content": "Write a haiku about cloud servers."}
    ]
  }'

To run the server as a systemd service for production, create /etc/systemd/system/gemma4-api.service:

[Unit]
Description=Gemma 4 FastAPI Server
After=network.target
[Service]
Type=simple
User=root
WorkingDirectory=/root/gemma4-project
Environment="PATH=/root/gemma4-project/venv/bin:/usr/bin"
ExecStart=/root/gemma4-project/venv/bin/uvicorn api_server:app --host 127.0.0.1 --port 8000
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable gemma4-api
sudo systemctl start gemma4-api

Method 3: Install via vLLM (High Performance)

vLLM is a high-throughput inference engine that uses PagedAttention for efficient memory management. It is the best choice when you need to serve many concurrent users, maximize tokens per second, or run a production API with high availability. vLLM requires a GPU.

Step 1: Install vLLM

Set up a Python environment and install vLLM:

mkdir -p ~/gemma4-vllm
cd ~/gemma4-vllm
python3 -m venv venv
source venv/bin/activate
pip install vllm

vLLM pulls in PyTorch with CUDA support automatically. The installation may take several minutes.

Step 2: Launch the OpenAI-Compatible Server

Start vLLM with the Gemma 4 model. vLLM downloads the model from Hugging Face automatically if it is not already cached:

export HUGGING_FACE_HUB_TOKEN="your-hf-token-here"
vllm serve google/gemma-4-12b-it \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16 \
  --api-key your-secret-api-key

Key flags explained:

• --max-model-len 32768 -- Maximum sequence length. Reduce this if you run low on VRAM. Increase up to 131072 if you have enough memory.
• --gpu-memory-utilization 0.90 -- Use up to 90% of GPU VRAM. Leave headroom for the OS and CUDA overhead.
• --dtype bfloat16 -- Use bfloat16 precision for best speed-to-quality ratio.
• --api-key -- Require an API key for all requests.

vllm serve google/gemma-4-27b-it \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16

Step 3: Test with curl

vLLM exposes the standard OpenAI API format:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-api-key" \
  -d '{
    "model": "google/gemma-4-12b-it",
    "messages": [
      {"role": "system", "content": "You are a senior DevOps engineer."},
      {"role": "user", "content": "Explain the difference between Docker and Podman."}
    ],
    "max_tokens": 512,
    "temperature": 0.7
  }'

You can also use the OpenAI Python SDK by pointing it to your vLLM server:

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-api-key",
)
response = client.chat.completions.create(
    model="google/gemma-4-12b-it",
    messages=[
        {"role": "user", "content": "What is Kubernetes?"}
    ],
)
print(response.choices[0].message.content)

Note: vLLM requires an NVIDIA GPU with at least 24 GB VRAM for the 12B model in bfloat16. For CPU-only servers, use Ollama (Method 1) with quantized models instead.

GPU Setup for Gemma 4

If your VPS has an NVIDIA GPU, you need the correct drivers and CUDA toolkit installed before running Gemma 4 at full speed.

Install NVIDIA Drivers

sudo apt install -y linux-headers-$(uname -r)
sudo apt install -y nvidia-driver-560

Reboot to load the driver:

sudo reboot

After reconnecting, verify the driver:

nvidia-smi

You should see your GPU model, driver version, and CUDA version in the output.

Install CUDA Toolkit

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-4

Add CUDA to your PATH:

echo 'export PATH=/usr/local/cuda-12.4/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Verify:

nvcc --version

Recommended GPU Configurations

Running Gemma 4 on CPU Only

Not every workload needs a GPU. If you are building a chatbot with moderate traffic, processing documents in batch overnight, or experimenting with the model, CPU inference with quantized weights is a viable option.

How Quantization Works

Quantization reduces model precision from 16-bit floats to 4-bit or 8-bit integers. This shrinks the model by 3-4x and allows it to fit in regular system RAM. The quality loss is minimal for most tasks -- quantized models typically retain 95-98% of the full-precision model's benchmark scores.

The most common quantization formats are:

• Q4_K_M -- 4-bit quantization with medium quality. Best balance of speed and quality. This is what Ollama uses by default.
• Q5_K_M -- 5-bit quantization. Slightly better quality, slightly more RAM.
• Q8_0 -- 8-bit quantization. Near-lossless quality, about 2x the size of Q4.

CPU Performance Expectations

Here is what to expect on typical VPS configurations running the Q4_K_M quantized 12B model:

• Internal tools where users wait a few seconds for a response
• Background batch processing (summarization, classification, extraction)
• Development and testing before deploying to a GPU server
• Low-traffic chatbots (under 10 concurrent users)

Running on CPU with Ollama

Ollama automatically uses CPU inference when no GPU is detected. No extra configuration is needed:

ollama run gemma4:12b

To explicitly request a quantized variant:

ollama pull gemma4:12b-q4_K_M
ollama run gemma4:12b-q4_K_M

Optimizing CPU Inference

Maximize CPU performance with these settings:

# Set the number of threads to your physical core count (not vCPU count)
export OLLAMA_NUM_THREADS=8
Restart Ollama to apply
sudo systemctl restart ollama

Ensure your VPS has adequate swap space as a safety net:

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Warning: Swap is a safety net, not a substitute for RAM. If the model actively uses swap during inference, performance degrades severely. Make sure your VPS has enough physical RAM for the quantized model plus 2-4 GB overhead.

Security and Access Control

Exposing a language model API to the internet without protection is dangerous. Anyone who finds your endpoint can run unlimited inference on your hardware, rack up compute costs, and potentially extract sensitive information from your model's context. Here is how to lock it down.

Set Up Nginx as a Reverse Proxy

Install Nginx:

sudo apt install -y nginx

Create a configuration file at /etc/nginx/sites-available/gemma4:

server {
    listen 80;
    server_name llm.yourdomain.com;
    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Increase timeouts for long-running inference requests
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
        proxy_connect_timeout 60s;
        # Increase body size for large prompts
        client_max_body_size 10m;
    }
}

Enable the site and restart Nginx:

sudo ln -s /etc/nginx/sites-available/gemma4 /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

Add SSL with Let's Encrypt

sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d llm.yourdomain.com

Certbot automatically configures Nginx for HTTPS and sets up certificate renewal.

Add API Key Authentication

Add a simple API key check in the Nginx configuration. Edit /etc/nginx/sites-available/gemma4 and add this inside the location / block:

# API key authentication
    set $api_key "your-long-random-api-key-here";
    if ($http_authorization != "Bearer $api_key") {
        return 401 '{"error": "Unauthorized"}';
    }

Clients must include the header Authorization: Bearer your-long-random-api-key-here with every request.

Configure the Firewall

Allow only SSH, HTTP, and HTTPS. Block direct access to the Ollama port:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Verify that port 11434 is not accessible from outside:

sudo ufw status

The Ollama service should only be reachable through the Nginx reverse proxy, which handles SSL termination and API key validation.

Performance Benchmarks

The following benchmarks were measured on VPS-Server.host plans running Ubuntu 24.04 with Ollama and Q4_K_M quantization (unless noted otherwise). Numbers represent sustained throughput, not burst performance.

Tokens Per Second by Model Size and Hardware

Concurrent Users by Configuration

Fine-Tuning Gemma 4 (Overview)

Gemma 4's open weights mean you can fine-tune it on your own data to improve performance on domain-specific tasks. The most practical approach for VPS-scale hardware is LoRA (Low-Rank Adaptation) or its memory-efficient variant QLoRA.

What Fine-Tuning Does

Fine-tuning adapts the model's behavior by training it on examples of your desired input-output pairs. Common use cases include:

• Training the model on your company's documentation so it answers support questions accurately.
• Teaching it your coding standards and API patterns for code generation.
• Adapting it to a specific writing style or tone for content generation.
• Improving accuracy on domain-specific tasks like medical coding, legal analysis, or financial reporting.

LoRA and QLoRA

LoRA freezes the original model weights and trains small adapter layers (typically 0.1-1% of the model's parameters). This means:

• Training VRAM: 12-16 GB for the 12B model (vs 50+ GB for full fine-tuning).
• Training time: A few hours on a single GPU for a small dataset.
• Storage: Adapter weights are typically 50-200 MB, not a full model copy.

Getting Started

The recommended tools for fine-tuning Gemma 4 are:

• Unsloth -- Optimized LoRA/QLoRA training with 2x speed improvement. Supports Gemma models natively. Install with pip install unsloth.
• Hugging Face TRL -- The SFTTrainer class provides a high-level interface for supervised fine-tuning with LoRA. Install with pip install trl peft.
• Axolotl -- A configuration-driven fine-tuning framework. Good for teams who prefer YAML configs over Python scripts.

Fine-tuning is a deep topic that deserves its own guide. For the full walkthrough, see our companion article: How to Fine-Tune Gemma 4 with LoRA on Ubuntu 24.04.

Troubleshooting

CUDA Out of Memory

Symptom: You see torch.cuda.OutOfMemoryError or CUDA error: out of memory when loading or running the model.

Solutions:

ollama pull gemma4:12b-q4_K_M

vllm serve google/gemma-4-12b-it --max-model-len 8192

ollama pull gemma4:4b

nvidia-smi

Slow Inference on CPU

Symptom: The model generates only 1-2 tokens per second.

Solutions:

nproc --all

htop

Model Download Fails

Symptom: ollama pull or huggingface-cli download fails partway through.

Solutions:

df -h

export HF_HUB_DOWNLOAD_TIMEOUT=600

Ollama Service Won't Start

Symptom: systemctl status ollama shows the service as failed.

Solutions:

journalctl -u ollama -n 50

sudo lsof -i :11434

sudo chown -R ollama:ollama /usr/share/ollama

Quantization Errors

Symptom: The model produces garbled or nonsensical output.

Solutions:

ollama rm gemma4:12b
   ollama pull gemma4:12b

ollama pull gemma4:12b-q5_K_M

sensors

FAQ

Is Gemma 4 better than Llama 3.1?

It depends on the task. Gemma 4 12B generally outperforms Llama 3.1 8B on multilingual tasks, code generation, and reasoning benchmarks. Llama 3.1 has broader ecosystem support due to its earlier release. For new deployments in 2025-2026, Gemma 4 is the stronger choice at equivalent parameter counts, particularly if you need multilingual or vision capabilities.

Can I use Gemma 4 commercially?

Yes. Gemma 4 is released under the Gemma license, which permits commercial use, fine-tuning, and redistribution. You can build products and services on top of Gemma 4 without paying royalties to Google. The license does include a responsible use policy that prohibits certain harmful applications, similar to Llama's acceptable use policy. Review the full license text at https://ai.google.dev/gemma/terms before deploying in a commercial setting.

What is the minimum RAM to run Gemma 4?

The 2B model runs with as little as 4 GB of RAM using Q4 quantization. The 12B model needs at least 16 GB (10 GB for the model plus overhead). The 27B model requires 32 GB minimum. These are minimums -- for comfortable operation with headroom for the OS and other services, add 4-8 GB to each figure. For the recommended VPS plans and exact memory requirements, see the Model Variants table.

Can I fine-tune Gemma 4 on my own data?

Yes. Gemma 4 supports fine-tuning with LoRA, QLoRA, and full-parameter methods. The most practical approach on VPS hardware is QLoRA, which lets you fine-tune the 12B model on a single GPU with 16 GB VRAM. You need a dataset of input-output pairs in your target domain -- typically 500-5,000 examples is enough to see meaningful improvement. See the Fine-Tuning section for details.

Should I use Gemma 4 or the Gemini API?

Use the Gemini API if you want zero infrastructure overhead, need Gemini Ultra-class capabilities, or have low and unpredictable usage. Use self-hosted Gemma 4 if you need data privacy, predictable costs at scale, no rate limits, offline capability, or the ability to fine-tune. Many teams use both: Gemini API for prototyping and complex tasks, Gemma 4 self-hosted for high-volume production inference.

Can I run Gemma 4 without a GPU?

Yes. The quantized models (Q4_K_M) run on CPU-only servers. Expect 5-8 tokens per second on an 8 vCPU machine with the 12B model, which translates to a 25-40 second wait for a typical 200-token response. This is perfectly usable for internal tools, batch processing, and low-traffic applications. For real-time applications with multiple concurrent users, a GPU significantly improves the experience. See Running Gemma 4 on CPU Only for details.

Next Steps

Now that Gemma 4 is running on your VPS, here are the natural next steps to build a complete self-hosted AI stack:

Install Open WebUI

Open WebUI gives you a ChatGPT-like web interface for your self-hosted models. It connects directly to your Ollama instance:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Access it at http://your-server-ip:3000. It supports multi-user accounts, conversation history, file uploads, and model switching.

Build a RAG Pipeline

Retrieval-Augmented Generation connects Gemma 4 to your own knowledge base. Upload PDFs, documentation, or database exports, and the model answers questions using your data as context. Popular frameworks for RAG include:

• LangChain -- Python framework with built-in Ollama integration
• LlamaIndex -- Specialized for document indexing and retrieval
• Haystack -- Production-grade RAG pipeline framework by deepset

Deploy Multiple Models

Ollama makes it easy to run multiple models simultaneously. Pull additional models for comparison or specialized tasks:

ollama pull codestral:22b     # Specialized code model
ollama pull llama3.1:8b       # Meta's model for comparison
ollama pull nomic-embed-text  # Embedding model for RAG

Each model loads into memory on demand and unloads after the configured keep-alive period.

Monitor and Scale

For production deployments, add monitoring with Prometheus and Grafana to track inference latency, token throughput, memory usage, and error rates. When a single server reaches its limit, deploy behind a load balancer with multiple inference nodes.

Deploy Gemma 4 on a GPU Server Today

Running the larger Gemma 4 models -- 12B at full precision or 27B in any format -- requires dedicated GPU hardware. Our GPU-accelerated VPS plans come with NVIDIA A4000 and A100 GPUs, pre-installed CUDA drivers, and NVMe storage fast enough to load models in seconds.

GPU Server plans start at EUR 89.99/mo with:

• NVIDIA A4000 (16 GB VRAM) or A100 (40/80 GB VRAM)
• 16-64 vCPU, 64-256 GB RAM
• 500 GB - 2 TB NVMe SSD
• Unmetered bandwidth
• Full root access with Ubuntu 24.04

Not sure which plan fits your workload? Contact our team and we will recommend the right configuration for your model size and traffic volume.