How to Install llama.cpp Server on Ubuntu 24.04 VPS: CPU-Friendly GGUF LLM Inference

If you want to run large language models on a plain, affordable VPS without a GPU, llama.cpp is the shortest path from source code to production. It is a C/C++ inference engine for GGUF-quantized models that squeezes remarkable performance out of ordinary x86 CPUs, exposes an OpenAI-compatible HTTP API, and ships with a lightweight built-in web UI. This guide walks you through installing llama.cpp on Ubuntu 24.04 from a clean VPS to a hardened production deployment behind Nginx with TLS.

Prefer a turnkey AI runtime? Consider Ollama for a simpler CLI-driven experience, or vLLM if you have a GPU and need maximum throughput. llama.cpp is the right choice when you want raw control and CPU-only deployment.

Table of Contents

What is llama.cpp?

llama.cpp is an open-source inference engine written in portable C/C++ by Georgi Gerganov. It was originally built to run Meta's LLaMA weights on a MacBook, but it has since grown into a full multi-architecture runtime that supports dozens of model families -- Llama 3, Mistral, Mixtral, Gemma, Phi-3, Qwen, DeepSeek, Command R, Yi, and many more -- via the GGUF (GPT-Generated Unified Format) binary model file.

At its core, llama.cpp provides three things:

Because llama.cpp is written in low-level C++ with hand-tuned SIMD kernels (AVX2, AVX-512, ARM NEON), it runs well on commodity CPUs -- no CUDA, no PyTorch, no Python. The same codebase also compiles with CUDA (NVIDIA GPUs), Metal (Apple Silicon), Vulkan (cross-vendor GPU), ROCm/HIP (AMD), and SYCL (Intel), so you can move the exact same GGUF file between machines.

For VPS users who do not have GPU access but still want to run real LLMs on real workloads -- chatbots, summarization, classification, RAG -- llama.cpp hits the sweet spot between performance and portability.

Why Self-Host on a CPU VPS?

Running an LLM on a CPU-only VPS is no longer a compromise -- for many use cases, it is the pragmatic production choice:

• Low fixed cost -- A 6 vCPU / 12 GB RAM VPS runs a quantized 7B-9B model comfortably for around EUR 20/month flat. Equivalent OpenAI API usage at production volume is typically 5-20x more.
• No GPU lock-in -- You are not waiting in a GPU availability queue. CPU-only hosts are plentiful, cheap, and available in every region.
• Full data privacy -- Prompts, documents, and responses never leave your server. Ideal for GDPR/HIPAA, legal, medical, and financial workloads.
• No per-token metering -- Batch summarize a million records overnight; the bill does not change.
• Predictable latency -- Latency is dominated by token generation speed (6-15 tok/s on a modern CPU for a 7B Q4 model), not network round-trips.
• Works offline -- Once the GGUF is downloaded, inference does not require any internet connectivity.
• Model choice is unlimited -- Any GGUF on Hugging Face -- including community fine-tunes, merges, and research models -- runs on the same binary.

CPU vs GPU vs Cloud API at a glance

Prerequisites

Before starting, make sure you have:

• A VPS running Ubuntu 24.04 LTS with root or sudo access.
• SSH access (terminal on macOS/Linux, PuTTY or Windows Terminal on Windows).
• At least 8 GB of RAM for 7B-class models (12 GB+ strongly recommended so you have headroom for context).
• At least 20 GB of free disk space (a quantized 7B model is ~4-5 GB; keep room for 2-3 of them).
• A modern x86-64 CPU with AVX2 (any VPS from the last ~8 years qualifies). AVX-512 is a bonus.

Recommended Plan: CloudCore Professional

For running 7B-9B models quantized at Q4 with a 4-8k context window, we recommend the CloudCore Professional plan:

- 6 vCPU cores (AVX2 + AVX-512)

- 12 GB RAM

- 100 GB NVMe SSD

- Unmetered bandwidth

- EUR 19.99/month

The NVMe is important -- llama.cpp uses mmap by default, so model load time depends directly on sequential read speed. You can drop to a smaller plan for 2B-3B models (Phi-3 Mini, Gemma 2 2B) or step up to 24 GB for 13B models. For 70B-class models, use a GPU plan and pair with vLLM.

Connect via SSH:

ssh root@your-server-ip

Step 1: Update System Packages

Start with a clean, patched system. This avoids compiler mismatches later.

sudo apt update && sudo apt upgrade -y

If the kernel was updated, reboot:

sudo reboot

Reconnect after a minute.

Step 2: Install Build Dependencies

llama.cpp builds with CMake and a C++ compiler. On Ubuntu 24.04 the default g++ is GCC 13, which is perfectly modern. You also need git, curl, and a few libraries used by the server component.

sudo apt install -y \
  build-essential \
  cmake \
  git \
  curl \
  wget \
  pkg-config \
  libcurl4-openssl-dev \
  ccache

Verify the toolchain:

cmake --version
g++ --version

Expected output (abbreviated):

cmake version 3.28.3
g++ (Ubuntu 13.2.0-23ubuntu4) 13.2.0

Optional: CUDA, Vulkan, or Metal

llama.cpp has pluggable backends selected at configure time. Pick one of the following depending on your hardware:

• CPU only (default, what this guide uses): no extra packages needed.
• NVIDIA CUDA: install the CUDA toolkit with sudo apt install -y nvidia-cuda-toolkit (or download a newer toolkit from NVIDIA), then pass -DGGML_CUDA=ON to CMake in the next step.
• Vulkan (cross-vendor GPU, including Intel Arc and older NVIDIA/AMD): sudo apt install -y libvulkan-dev glslang-tools then pass -DGGML_VULKAN=ON.
• Apple Silicon Metal: not relevant on a Linux VPS; this only applies when building on macOS. If you are on an M-series Mac, CMake enables Metal automatically.
• AMD ROCm/HIP: install the ROCm stack from AMD's repository and use -DGGML_HIP=ON.

Step 3: Clone the llama.cpp Repository

Create a workspace and clone the upstream repository from GitHub:

sudo mkdir -p /opt/llamacpp
sudo chown "$USER":"$USER" /opt/llamacpp
cd /opt/llamacpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

The repository is active -- new model architectures land almost daily. Pin to a release tag for production reproducibility:

git fetch --tags
git checkout $(git describe --tags $(git rev-list --tags --max-count=1))

This checks out the latest tagged release rather than master. If you want bleeding-edge support for a brand-new model family, stay on master instead.

Step 4: Build llama.cpp with CMake

llama.cpp uses an out-of-source CMake build. Create a build directory, configure, and compile.

CPU build (default)

cd /opt/llamacpp/llama.cpp
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON \
  -DGGML_NATIVE=ON
cmake --build build --config Release -j"$(nproc)"

Flag reference:

• -DCMAKE_BUILD_TYPE=Release -- enables compiler optimizations (-O3). Without it you will get a debug build that is 5-10x slower.
• -DLLAMA_CURL=ON -- links libcurl into llama-server so you can pass a Hugging Face URL or -hf repo identifier directly.
• -DGGML_NATIVE=ON -- compiles with -march=native, enabling every SIMD instruction your CPU supports (AVX2, AVX-512, FMA, F16C). This is the single biggest performance knob on CPU. Only set this if the binary will run on the same machine it was compiled on (different CPU models may not support the same instruction set).

CUDA build (NVIDIA GPU VPS)

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON \
  -DGGML_CUDA=ON
cmake --build build --config Release -j"$(nproc)"

Vulkan build

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=ON \
  -DGGML_VULKAN=ON
cmake --build build --config Release -j"$(nproc)"

Verify the binaries

ls build/bin/ | head -20
./build/bin/llama-server --version

Expected:

llama-batched
llama-bench
llama-cli
llama-embedding
llama-perplexity
llama-quantize
llama-server
llama-tokenize
...
version: 3847 (abcd1234)
built with gcc (Ubuntu 13.2.0-23ubuntu4) for x86_64-linux-gnu

The two binaries you will use most are llama-cli (one-shot / interactive CLI) and llama-server (HTTP API).

Install the server binary to /usr/local/bin so it is on your PATH:

sudo cp build/bin/llama-server /usr/local/bin/llama-server
sudo cp build/bin/llama-cli /usr/local/bin/llama-cli
sudo cp build/bin/llama-quantize /usr/local/bin/llama-quantize

Step 5: Download a GGUF Model from Hugging Face

llama.cpp loads models from a single .gguf file. The Hugging Face Hub hosts thousands of pre-quantized GGUF conversions, most published by TheBloke, bartowski, lmstudio-community, and QuantFactory.

We will use Meta Llama 3.1 8B Instruct, quantized to Q4_K_M (the recommended quality/size sweet spot). Create a models directory first:

sudo mkdir -p /var/lib/llamacpp/models
sudo chown "$USER":"$USER" /var/lib/llamacpp/models
cd /var/lib/llamacpp/models

Download the file with wget:

wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

The file is approximately 4.9 GB and takes 1-3 minutes on a typical VPS uplink.

Alternatively, if you built with LLAMA_CURL=ON, llama-server accepts a Hugging Face repo shorthand and downloads the model on first launch:

llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M

Choosing a model

For a 12 GB CPU VPS, these are solid choices:

Step 6: Run llama-server

Start the HTTP server with your downloaded model:

llama-server \
  -m /var/lib/llamacpp/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -c 4096 \
  -t $(nproc) \
  --mlock

Flag reference:

• -m -- path to the GGUF file.
• --host 127.0.0.1 -- bind to localhost only. We will put Nginx in front of it in Step 11.
• --port 8080 -- HTTP port.
• -c 4096 -- context window in tokens (prompt + response). See Step 8.
• -t $(nproc) -- number of CPU threads. Setting this to the physical core count usually gives the best performance.
• --mlock -- pins the model in RAM so the kernel cannot swap it out. Use only if you have enough RAM; otherwise the process will fail to start.

llama_model_loader: loaded meta data with 29 key-value pairs and 292 tensors
llm_load_print_meta: model type       = 8B
llm_load_print_meta: model params     = 8.03 B
llm_load_print_meta: model size       = 4.58 GiB (4.89 BPW)
llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: KV self size = 512.00 MiB
main: HTTP server listening on 127.0.0.1:8080

Leave it running in this terminal for now. Open a second SSH session to run the test calls in the next step.

Step 7: Test the OpenAI-Compatible API

llama-server implements the OpenAI v1 chat schema, so any client library that talks to OpenAI works unmodified -- just change the base URL.

Plain completion (curl)

curl http://127.0.0.1:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain what a reverse proxy is in two sentences.",
    "n_predict": 128,
    "temperature": 0.3
  }'

OpenAI-compatible chat endpoint

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [
      {"role": "system", "content": "You are a concise Linux sysadmin."},
      {"role": "user", "content": "How do I check disk usage on Ubuntu?"}
    ],
    "temperature": 0.2,
    "max_tokens": 256
  }'

Expected response (abbreviated):

{
  "id": "chatcmpl-abcd1234",
  "object": "chat.completion",
  "created": 1744800000,
  "model": "llama-3.1-8b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Use df -h for filesystem-level usage and du -sh <dir> for a specific directory. The -h flag shows human-readable sizes."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 42,
    "completion_tokens": 38,
    "total_tokens": 80
  }
}

Python with the OpenAI SDK

pip install openai

from openai import OpenAI
client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="not-needed",
)
resp = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Summarize the benefits of a VPS in 3 bullets."}],
)
print(resp.choices[0].message.content)

The api_key is ignored by llama-server unless you configured one with --api-key, but the SDK requires the parameter to be set.

Streaming responses

Add "stream": true and read SSE chunks:

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role": "user", "content": "Count from 1 to 5."}],
    "stream": true
  }'

Each line prefixed with data: is a token delta. The stream terminates with data: [DONE].

Step 8: Tune Context Length, mmap, and NUMA

Three flags have an outsized impact on memory, latency, and throughput.

Context length (-c)

The -c flag sets the size of the KV cache in tokens. Every token in the prompt and every generated token must fit. Memory cost is linear and, for a 7B model at Q4, roughly:

Memory mapping (--mmap / --no-mmap)

By default llama.cpp uses mmap() to load the model. This has several benefits:

• Faster startup -- the OS lazily pages in weights as they are read, so llama-server is responsive within a second.
• Shared memory -- multiple processes mapping the same GGUF share physical pages.
• Lower initial RAM -- pages are evicted under memory pressure.

llama-server -m model.gguf --mlock

Disable mmap with --no-mmap if you are on a filesystem that does not support it well (some network-mounted volumes), at the cost of slower startup and higher initial RAM use.

NUMA on multi-socket servers (--numa)

If your VPS has more than one CPU socket (rare on cloud VPS, common on dedicated/bare-metal), NUMA-aware placement helps a lot. Three policies:

# Distribute threads across nodes (good default for multi-socket)
llama-server -m model.gguf --numa distribute
Isolate on a single node
llama-server -m model.gguf --numa isolate
Use numactl interleave (manual)
numactl --interleave=all llama-server -m model.gguf --numa numactl

On a single-socket VPS (typical for CloudCore Professional), NUMA tuning is a no-op -- leave it off.

Step 9: Understand Quantization (Q2_K to Q8_0)

Quantization is the compression technique that makes CPU inference practical. Instead of storing each weight as a 16-bit or 32-bit float, GGUF stores weights in 2-8 bits using block-wise schemes. Lower bits = smaller files and less RAM, at the cost of some quality.

The names you will see on Hugging Face:

Quantizing a model yourself

If you have an F16 GGUF but need a smaller quant (or one that is not published), use the bundled llama-quantize tool:

llama-quantize \
  /var/lib/llamacpp/models/model-f16.gguf \
  /var/lib/llamacpp/models/model-Q4_K_M.gguf \
  Q4_K_M

This runs on CPU, uses little RAM, and takes 1-5 minutes depending on model size.

Step 10: Create a systemd Service

Running llama-server in a terminal is fine for testing, but for production you want it managed by systemd so it restarts on failure, starts on boot, and runs under an unprivileged user.

Create a dedicated user

sudo useradd --system --home /var/lib/llamacpp --shell /usr/sbin/nologin llamacpp
sudo chown -R llamacpp:llamacpp /var/lib/llamacpp

Write the unit file

sudo tee /etc/systemd/system/llamacpp.service > /dev/null <<'EOF'
[Unit]
Description=llama.cpp Server (OpenAI-compatible LLM API)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=llamacpp
Group=llamacpp
WorkingDirectory=/var/lib/llamacpp
ExecStart=/usr/local/bin/llama-server \
  -m /var/lib/llamacpp/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -c 4096 \
  -t 6 \
  --mlock \
  --metrics
Restart=on-failure
RestartSec=5
LimitMEMLOCK=infinity
LimitNOFILE=65536
Hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/llamacpp
[Install]
WantedBy=multi-user.target
EOF

Key points:

• LimitMEMLOCK=infinity -- required for --mlock to work under systemd.
• -t 6 -- match to your vCPU count (adjust on other plans).
• --metrics -- exposes Prometheus-style metrics at /metrics for monitoring.
• Hardening directives limit filesystem access to the model directory only.

Enable and start

sudo systemctl daemon-reload
sudo systemctl enable --now llamacpp
sudo systemctl status llamacpp

Expected:

llamacpp.service - llama.cpp Server (OpenAI-compatible LLM API)
     Loaded: loaded (/etc/systemd/system/llamacpp.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-04-16 12:00:00 UTC; 5s ago
   Main PID: 2345 (llama-server)
      Tasks: 8 (limit: 14236)
     Memory: 5.1G
        CPU: 12.3s

Tail logs:

sudo journalctl -u llamacpp -f

Step 11: Put llama-server Behind Nginx

Exposing port 8080 directly is fine for local development but unacceptable for production. Place Nginx in front for TLS termination, HTTP basic auth, rate limiting, and streaming-safe buffering. If you have not installed Nginx yet, follow our Nginx install guide first.

Install Nginx and htpasswd:

sudo apt install -y nginx apache2-utils

Create an API key (basic auth) file:

sudo htpasswd -c /etc/nginx/.llamacpp-htpasswd apiuser

Write the site configuration:

sudo tee /etc/nginx/sites-available/llamacpp > /dev/null <<'EOF'
server {
    listen 443 ssl http2;
    server_name llm.example.com;
    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options SAMEORIGIN;
    add_header Strict-Transport-Security "max-age=31536000" always;
    # Allow long prompts
    client_max_body_size 10m;
    location / {
        auth_basic "llama.cpp API";
        auth_basic_user_file /etc/nginx/.llamacpp-htpasswd;
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Connection "";
        # Streaming-friendly
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
        chunked_transfer_encoding on;
    }
    # Metrics endpoint, restrict to your monitoring IP
    location = /metrics {
        allow 10.0.0.0/8;
        deny all;
        proxy_pass http://127.0.0.1:8080/metrics;
    }
}
server {
    listen 80;
    server_name llm.example.com;
    return 301 https://$host$request_uri;
}
EOF

Enable the site and issue a certificate:

sudo ln -s /etc/nginx/sites-available/llamacpp /etc/nginx/sites-enabled/
sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d llm.example.com
sudo nginx -t && sudo systemctl reload nginx

Test with basic auth:

curl -u apiuser:yourpassword https://llm.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role": "user", "content": "Hello from Nginx!"}]
  }'

The proxy_buffering off directive is critical -- without it Nginx buffers the entire response and streaming ("stream": true) will not work.

Step 12: Use the Built-in Web UI

llama-server ships with a lightweight chat UI at the root path. After you point a browser at your server, you get a ChatGPT-style interface with model selection, temperature controls, and chat history.

Open:

https://llm.example.com/

You will be prompted for the basic-auth credentials you set up, then land on the chat UI. It is purely front-end (static files served by llama-server), so it works without any separate frontend deployment. For a richer multi-user interface, you can pair llama.cpp with Open WebUI pointed at the OpenAI-compatible endpoint, or LibreChat, or any OpenAI-compatible client.

Performance Tuning

Once you have a working server, these knobs give you 20-50% more throughput on CPU.

Match threads to physical cores

--threads 6          # prompt + generation threads
--threads-batch 6    # prompt processing threads (defaults to --threads)

Hyperthreading (SMT) usually hurts llama.cpp throughput -- pin to physical cores only. nproc reports logical cores, so divide by 2 on SMT CPUs, or check lscpu | grep "Core(s) per socket".

Batch size for prompt processing

--batch-size 512
--ubatch-size 128

Larger batches make prompt prefill faster (fewer SIMD kernel calls) but use more RAM. Defaults are usually fine; raise --batch-size to 1024-2048 if your prompts are consistently long.

Parallel slots (serving multiple users)

--parallel 4
-c 16384     # total context, split across slots -> 4096 per slot

--parallel N lets llama-server serve N independent conversations concurrently. The context is divided, so you need to raise -c accordingly. Concurrency on CPU is capped by memory bandwidth, so the practical ceiling is 2-4 parallel slots on a typical VPS.

Cache the system prompt

--system-prompt-file system.txt

If every request shares a long system prompt, caching it avoids re-processing on each call -- a big latency win for agent workloads.

Benchmark

llama-bench -m /var/lib/llamacpp/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

This reports prompt-processing and token-generation tokens/sec. On a 6 vCPU Xeon with AVX-512 and Q4_K_M, expect ~20 tok/s generation and ~120 tok/s prompt processing.

Troubleshooting

Viewing logs

# Service logs
sudo journalctl -u llamacpp -n 100 --no-pager
Follow live
sudo journalctl -u llamacpp -f

Health check

curl -s http://127.0.0.1:8080/health
{"status":"ok","slots_idle":1,"slots_processing":0}

FAQ

How does llama.cpp compare to Ollama and vLLM?

Ollama is a polished wrapper around llama.cpp that adds CLI ergonomics, model management (ollama pull), and a daemon architecture. If you want "Docker for LLMs", start with Ollama. llama.cpp gives you direct control over every flag (quantization, NUMA, batch sizes, chat templates, sampling parameters), is easier to embed in other applications as a library (libllama.so), and compiles into a single static binary with no daemon. vLLM is a different beast entirely -- a GPU-first, Python-based inference server built around PagedAttention and continuous batching. It is the right answer for high-concurrency GPU deployments; see the vLLM install guide. For CPU-only production, llama.cpp is the go-to.

Can I run llama.cpp without a GPU?

Yes -- that is its flagship use case. Modern CPUs with AVX2 or AVX-512 generate 7B-model tokens at 6-20 per second, which is perfectly serviceable for chatbots, back-office tooling, summarization, classification, and RAG. GPU acceleration is a drop-in (-DGGML_CUDA=ON), but there is no penalty for staying on CPU other than throughput.

What is GGUF and why do I need it?

GGUF is the binary model format used by llama.cpp (the successor to the older GGML format). It contains the quantized weights, tokenizer, and metadata in a single mmap-friendly file. Hugging Face hosts thousands of pre-converted GGUFs from maintainers like bartowski and TheBloke. If you only find safetensors files, use the convert_hf_to_gguf.py script in the llama.cpp repo to convert and then llama-quantize to compress.

How much RAM do I actually need?

RAM = model file size + KV cache + ~500 MB overhead. For Llama 3.1 8B at Q4_K_M with a 4k context, that is roughly 4.9 + 0.5 + 0.5 = ~6 GB. Add 2-3 GB of headroom for the OS, your application, and buffers. A 12 GB VPS like CloudCore Professional runs this comfortably with room for a 16k context window.

Can I serve multiple users concurrently?

Yes, with the --parallel N flag. Each slot gets its own KV cache, so total memory scales with slot count. On CPU, the practical ceiling is 2-4 concurrent generations before memory bandwidth becomes the bottleneck -- but that is plenty for most internal tools, chatbots, and low-volume SaaS backends.

How do I integrate with LangChain or LlamaIndex?

Both frameworks have first-class llama.cpp support via the OpenAI-compatible endpoint. Just point the OpenAI client at http://127.0.0.1:8080/v1 with any dummy API key. For LangChain: ChatOpenAI(base_url="...", api_key="x"). For LlamaIndex: OpenAI(api_base="...", api_key="x"). No llama.cpp-specific bindings needed.

How do I update llama.cpp?

The codebase evolves fast. Pull the latest tag and rebuild:

cd /opt/llamacpp/llama.cpp
git fetch --tags
git checkout $(git describe --tags $(git rev-list --tags --max-count=1))
cmake --build build --config Release -j"$(nproc)"
sudo cp build/bin/llama-server /usr/local/bin/llama-server
sudo systemctl restart llamacpp

Breaking changes to the CLI or server API are rare but do happen -- skim the release notes before upgrading in production.

Next Steps

Your llama.cpp server is live, TLS-protected, and ready for real traffic. A few directions to grow from here:

• Add a richer chat UI -- Pair llama.cpp with Open WebUI or LibreChat for multi-user chat history, role-based access, and a modern interface.
• Build a RAG pipeline -- Use a vector store like Qdrant or ChromaDB, generate embeddings with llama-embedding (or a dedicated embedding model like nomic-embed-text in GGUF), and feed context into chat completions.
• Scale to GPU -- When CPU throughput is no longer enough, rebuild with CUDA or move to vLLM on a GPU VPS. The GGUF ecosystem and OpenAI API shape let you swap runtimes without changing application code.
• Monitor with Prometheus -- llama-server's /metrics endpoint exposes request counters and timing. Scrape it with Prometheus and graph in Grafana alongside system metrics.
• Experiment with fine-tunes -- Community fine-tunes on Hugging Face (Nous Hermes, Dolphin, OpenHermes, etc.) are almost always available in GGUF. Swap the -m flag to try a new one in seconds.
• Harden with Fail2Ban -- Protect the basic-auth endpoint from credential stuffing by configuring Fail2Ban to watch Nginx 401 responses.

Need a CPU VPS sized for llama.cpp?

The CloudCore Professional plan gives you 6 vCPU, 12 GB RAM, and 100 GB NVMe for EUR 19.99/month -- enough for a 7B-9B quantized model at 4-8k context with headroom to spare. Deploy in 60 seconds and follow this guide end to end.

Deploy CloudCore Professional now