GUIDEInstall Guides

How to Build a Self-Hosted Monitoring Stack (Prometheus + Grafana + Loki) on Ubuntu 24.04

33 min read

How to Build a Self-Hosted Monitoring Stack (Prometheus + Grafana + Loki) on Ubuntu 24.04

Observability used to be the domain of whoever could stomach a Datadog invoice. A dozen hosts, a handful of containers, and a reasonable log volume and you are staring at a four-figure monthly bill that climbs every time a developer ships a feature. The good news: the open-source tooling has quietly become so good that a single VPS can run a complete production-grade monitoring stack — metrics, logs, dashboards, and alerting — for less than what Datadog charges for one host.

This guide walks you through the full build on Ubuntu 24.04: Prometheus for metrics, Grafana for visualisation, Loki for logs, Promtail as the log shipper, Alertmanager for routing alerts, and Node Exporter plus cAdvisor for collecting host and container telemetry. By the end you will have a hardened, HTTPS-terminated dashboard at grafana.example.com, automated alerts going to Slack and email, a month of retained logs, and a backup story that lets you rebuild the stack in fifteen minutes if the VPS melts.

Want to skip provisioning? Our CloudCore Business plan comes with Docker pre-installed and enough headroom for this entire stack. Launch a VPS in 60 seconds and follow along.

Architecture Overview

Why Self-Host Observability?

Cost Comparison: Self-Hosted vs Datadog

Prerequisites and VPS Sizing

The Components Explained

Step 1: Prepare the Server

Step 2: Directory Layout and Environment File

Step 3: Prometheus Configuration

Step 4: Alertmanager Configuration

Step 5: Loki and Promtail Configuration

Step 6: Grafana Provisioning

Step 7: The Complete docker-compose.yml

Step 8: Launch the Stack

Step 9: Import Dashboards and Validate

Step 10: Alert Rules and LogQL Examples

Step 11: nginx Reverse Proxy with SSL

Step 12: Backups

Scaling Considerations

Troubleshooting

FAQ

Next Steps

Architecture Overview

The stack is seven cooperating containers behind a single nginx reverse proxy. Grafana is the only one exposed to the public internet; everything else stays on a private Docker network or bound to 127.0.0.1.

Internet
                                 |
                                 | HTTPS (443)
                                 v
                       +---------------------+
                       |        nginx        |   <- Let's Encrypt SSL
                       |    (host network)   |
                       +---------------------+
                                 |
                          grafana.example.com
                                 v
                          +--------------+
                          |   Grafana    |  (127.0.0.1:3000)
                          +--------------+
                            |         |
                 datasource |         | datasource
                            v         v
                   +-------------+  +-------------+
                   |  Prometheus |  |    Loki     |
                   | (9090, int) |  | (3100, int) |
                   +-------------+  +-------------+
                     ^   ^    |            ^
            scrape   |   |    | rules      | push
                     |   |    v            |
            +--------+   |  +---------------+
            |            |  | Alertmanager  |
            |            |  | (9093, int)   |--> Slack / email / PagerDuty
            |            |  +---------------+
            |            |
     +------+------+  +--+---------+  +-----------+
     | Node Exporter|  |  cAdvisor  |  |  Promtail |
     |   (9100)     |  |   (8080)   |  |  (agent)  |
     +--------------+  +------------+  +-----------+
                                             ^
                                             | tail
                                             |
                                       /var/log/*
                                       docker logs

Data flow for metrics:

Node Exporter reads /proc, /sys, and / on the host and exposes 1,000+ metrics as Prometheus text on :9100/metrics.

cAdvisor walks the Docker runtime and exposes per-container CPU, memory, network, and IO metrics on :8080/metrics.

Prometheus scrapes both endpoints every 15 seconds, stamps each sample with labels, and writes them to its local TSDB.

Alertmanager continuously evaluates Prometheus alerting rules and, when a rule fires, routes the alert to Slack, email, or PagerDuty according to its routing tree.

Grafana queries Prometheus (and Loki) live via PromQL and LogQL when you open a dashboard.

Data flow for logs:

Promtail runs as a container with /var/log, /var/lib/docker/containers, and the Docker socket bind-mounted read-only.

It tails every log file, parses the labels (service, level, container name), batches the lines, and pushes them to Loki over HTTP.

Loki stores the compressed chunks on local disk (or S3/GCS for production) indexed by label, not by full text — this is why Loki is cheap to run.

Grafana's Explore view queries Loki with LogQL: {container="nginx"} |= "error" returns lines live.

Everything persists to named Docker volumes so you can upgrade individual containers without losing history.

Why Self-Host Observability?

Predictable flat-rate pricing. SaaS observability is charged per host, per metric, per log GB, per custom metric, per synthetic check, and per user. Self-hosted is one VPS line item that does not move when you add a service or double your log volume.

No cardinality tax. Datadog charges extra for "custom metrics" which in practice means anything with a user-defined label. Prometheus gives you unlimited labels for free; the only limit is your VPS RAM. Label everything — tenant_id, customer_tier, feature_flag — without watching a meter spin.

Real data ownership. Metrics and logs often contain PII, request payloads, customer identifiers, and trade secrets. Keeping them on a box you control removes the entire class of "vendor breach leaks our customer data" incident. For GDPR and HIPAA-regulated operators this is less a preference than a requirement.

Integration freedom. Prometheus is the de-facto metrics standard. Every modern database, message queue, web server, and SaaS has a Prometheus exporter. LogQL's syntax is familiar to anyone who has used PromQL. Grafana speaks to over sixty data sources out of the box — including your existing CloudWatch, BigQuery, Elasticsearch, or Datadog account if you want a hybrid setup.

Operational depth. When your stack misbehaves you can docker exec into Prometheus and look at the raw TSDB blocks, tail Loki's ingester logs, and rewrite rules on the fly. With a SaaS vendor you open a ticket and wait.

Learning and control. Standing the stack up teaches you the mechanics of modern observability — exposition formats, scrape intervals, label cardinality, push vs pull, chunk-based log stores — in a way that clicking around Datadog's UI never will.

Cost Comparison: Self-Hosted vs Datadog

Assume a modest fleet: ten Linux hosts, thirty containers, and 50 GB of log ingest per month. This is a middle-of-the-road startup footprint.

Datadog (cloud)

Line item	Monthly cost
Pro hosts (10 × $15/host)	$150
Pro APM (10 × $31/host)	$310
Log ingest (50 GB × $0.10/GB)	$5
Log retention 30 days (50 GB × $1.70/GB)	$85
Custom metrics (200 × $0.05)	$10
Synthetic API tests (10k/mo)	$15
Total	~$575/mo

Datadog's list pricing actually runs $31-70/host/month once you include APM, live processes, DBM, and the log retention you actually need. Enterprise quotes for this fleet size routinely land at $700-1,200/mo.

Self-hosted stack (CloudCore Business 4 vCPU / 8 GB / 80 GB NVMe)

Line item	Monthly cost
VPS (CloudCore Business)	EUR 35.99
Domain (amortized)	EUR 1
Backup storage (off-site)	EUR 3
Total	~EUR 40/mo (~$43/mo)

That is a 92% reduction at the same usage envelope. Add another ten hosts and the Datadog bill nearly doubles; the self-hosted bill does not move until you exhaust VPS RAM (typically around 30-50 monitored hosts) at which point you scale one tier.

Where SaaS still wins: managed service — no one pages you at 3 am because Prometheus filled its disk. Mitigations: alert on the stack itself (there is a section on this below), run weekly backups, and keep an infrastructure-as-code deployment so you can rebuild in fifteen minutes. For teams unwilling to run their own stack, a managed Grafana Cloud free tier covers up to three users and 10k active metrics — a reasonable hybrid.

The break-even point against Datadog is roughly two hosts. Past that, every additional host is pure savings.

Prerequisites and VPS Sizing

Minimum (up to 20 monitored hosts, light log volume):

2 vCPU
4 GB RAM
40 GB NVMe SSD
Ubuntu 24.04 LTS
Public IPv4
Domain with DNS access

Recommended (20-50 hosts, 50-200 GB logs/mo, 30-day retention):

4 vCPU
8 GB RAM
80 GB NVMe SSD
Ubuntu 24.04 LTS

Large (50-150 hosts, heavy logs, 90-day retention):

8 vCPU
16 GB RAM
240 GB NVMe SSD
External object storage for Loki chunks (Backblaze B2, S3)

These map directly to our CloudCore plans. The recommended tier is the sweet spot for small and mid-sized startups and is what the rest of this guide assumes.

You will also need: a domain, a registered Slack workspace (for alert webhooks) or an email relay, and basic SSH and Linux skills.

The Components Explained

Prometheus — the metrics engine. A pull-based time-series database written in Go. It scrapes HTTP endpoints at a configured interval, stores samples in a purpose-built TSDB, and exposes PromQL for querying. Prometheus is the standard everything else builds on.

Alertmanager — the alert router. Takes alerts fired by Prometheus and decides what to do with them: deduplicate, group, silence during maintenance windows, and send to one or more notifiers (Slack, email, PagerDuty, Opsgenie, webhooks). Critically, it sits separate from Prometheus so your alerting stays sane during Prometheus restarts.

Grafana — the dashboard layer. The industry-standard visualisation front-end. Talks to Prometheus, Loki, Postgres, CloudWatch, and dozens more. Ships a vast gallery of community dashboards you can import by ID. Also handles users, teams, folders, and (in OSS) alerting — though we defer to Alertmanager for the production alerting path.

Loki — logs, indexed by labels not text. Grafana Labs' answer to "what if logs worked like Prometheus". Instead of indexing every word (the ElasticSearch approach that costs a fortune), Loki only indexes the labels attached to each log stream. Text search happens via fast brute-force scans over compressed chunks. The result: one-tenth the storage and cost of an ELK stack for the same log volume.

Promtail — the log shipper. Loki's agent. Runs on every host, tails log files and Docker container logs, attaches labels (container, service, tenant), batches lines, and pushes them to Loki. Configured entirely in YAML with a relabel pipeline borrowed from Prometheus.

Node Exporter — host-level metrics. The canonical Prometheus exporter for Linux. Exposes CPU, memory, disk, network, filesystem, TCP, systemd unit state, and hundreds of other metrics on :9100/metrics. Install it on every host you want monitored.

cAdvisor — container-level metrics. Google's container advisor. Reads the Docker and cgroup APIs and exposes per-container CPU, memory, network, and IO on :8080/metrics. Combined with Node Exporter it gives you complete coverage of the host-plus-containers surface.

nginx — the TLS reverse proxy. Terminates HTTPS, routes grafana.example.com to the container, handles WebSocket upgrades for Grafana Live, and enforces a global rate limit.

Step 1: Prepare the Server

SSH in as a non-root sudo user and update:

sudo apt update && sudo apt upgrade -y
sudo apt install -y ca-certificates curl gnupg lsb-release ufw fail2ban git jq

Install Docker Engine and Compose v2 from Docker's official APT repo:

sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \ sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg sudo chmod a+r /etc/apt/keyrings/docker.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin sudo usermod -aG docker $USER newgrp docker

Configure the firewall — only 22, 80, and 443 open to the world:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Point a single DNS A record at the server's public IP:

grafana.example.com     A   203.0.113.42

Deliberately do not expose prometheus.example.com, loki.example.com, or alertmanager.example.com. Those services stay internal and you reach them through Grafana. This is not only a security posture — it is also the idiomatic way Prometheus is meant to be deployed.

Make sure the kernel time is synchronised — Prometheus's correctness depends on it:

sudo timedatectl set-ntp true
timedatectl status

Step 2: Directory Layout and Environment File

Create a clean working directory:

sudo mkdir -p /opt/monitoring
sudo chown $USER:$USER /opt/monitoring
cd /opt/monitoring
mkdir -p configs/{prometheus/rules,alertmanager,loki,promtail,grafana/provisioning/datasources,grafana/provisioning/dashboards,grafana/dashboards}
mkdir -p backups scripts

Create /opt/monitoring/.env:

# --- general ---
TZ=UTC
DOMAIN=example.com
--- Grafana ---
GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=REPLACE_WITH_openssl_rand_hex_24
GF_SERVER_ROOT_URL=https://grafana.example.com
GF_USERS_ALLOW_SIGN_UP=false
--- Alert notifiers ---
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/XXXXXXXX
SMTP_SMARTHOST=smtp.example.com:587
[email protected]
SMTP_PASSWORD=REPLACE
[email protected]
[email protected]
--- Retention ---
PROMETHEUS_RETENTION=30d
PROMETHEUS_RETENTION_SIZE=20GB
LOKI_RETENTION=720h

Generate the Grafana admin password:

echo "GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -hex 24)"

Paste into .env and lock it down:

chmod 600 /opt/monitoring/.env

Step 3: Prometheus Configuration

Create /opt/monitoring/configs/prometheus/prometheus.yml:

global: scrape_interval: 15s evaluation_interval: 15s external_labels: cluster: production replica: a rule_files: - /etc/prometheus/rules/*.yml alerting: alertmanagers: - static_configs: - targets: ['alertmanager:9093'] scrape_configs: # --- Prometheus itself --- - job_name: prometheus static_configs: - targets: ['localhost:9090'] # --- Host metrics via Node Exporter --- - job_name: node static_configs: - targets: ['node-exporter:9100'] labels: host: monitoring-vps # --- Container metrics via cAdvisor --- - job_name: cadvisor static_configs: - targets: ['cadvisor:8080'] # --- Alertmanager self-metrics --- - job_name: alertmanager static_configs: - targets: ['alertmanager:9093'] # --- Loki self-metrics --- - job_name: loki static_configs: - targets: ['loki:3100'] # --- Grafana self-metrics --- - job_name: grafana static_configs: - targets: ['grafana:3000']

# --- Remote Node Exporters on other hosts --- # Replace with your actual inventory. Wire up mTLS or a VPN # (Tailscale, WireGuard) before exposing :9100 across the public internet. - job_name: remote-nodes static_configs: - targets: - '10.0.0.11:9100' - '10.0.0.12:9100' - '10.0.0.13:9100' labels: env: production

Create an initial alerting rule file at /opt/monitoring/configs/prometheus/rules/node-alerts.yml:

groups:
  - name: node-basics
    interval: 30s
    rules:
      - alert: HostDown
        expr: up{job=~"node|remote-nodes"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Host {{ $labels.instance }} is down"
          description: "Prometheus has not scraped {{ $labels.instance }} for 2 minutes."
      - alert: HostHighCpu
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU has been above 90% for 10 minutes (current {{ $value | printf \"%.1f\" }}%)."
      - alert: HostHighMemory
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | printf \"%.1f\" }}% for 10 minutes."
      - alert: HostDiskFillingUp
        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) * 100 < 10
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Disk nearly full on {{ $labels.instance }} ({{ $labels.mountpoint }})"
          description: "Only {{ $value | printf \"%.1f\" }}% free on {{ $labels.mountpoint }}."
  - name: container-basics
    interval: 30s
    rules:
      - alert: ContainerRestarting
        expr: increase(container_start_time_seconds[15m]) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} is restart-looping"
          description: "{{ $labels.name }} has restarted more than 3 times in 15 minutes."      - alert: ContainerHighMemory
        expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100 > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} is at {{ $value | printf \"%.0f\" }}% of its memory limit"

Step 4: Alertmanager Configuration

Create /opt/monitoring/configs/alertmanager/alertmanager.yml:

global:
  resolve_timeout: 5m
  smtp_smarthost: "${SMTP_SMARTHOST}"
  smtp_from:      "${ALERT_FROM}"
  smtp_auth_username: "${SMTP_USER}"
  smtp_auth_password: "${SMTP_PASSWORD}"
  smtp_require_tls: true
route:
  receiver: default-slack
  group_by: ['alertname', 'cluster', 'severity']
  group_wait:      30s
  group_interval:  5m
  repeat_interval: 3h
  routes:
    - match:
        severity: critical
      receiver: critical-combo
      repeat_interval: 30m
      continue: true
    - match:
        severity: warning
      receiver: default-slack
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'instance']
receivers:
  - name: default-slack
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#alerts"
        send_resolved: true
        title: "[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}"
        text: |
          {{ range .Alerts }}
          {{ .Annotations.summary }}
          {{ .Annotations.description }}
          Severity: {{ .Labels.severity }}
          Instance: {{ .Labels.instance }}
          {{ end }}  - name: critical-combo
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#alerts-critical"
        send_resolved: true
        title: ":rotating_light: {{ .CommonLabels.alertname }}"
        text: |
          {{ range .Alerts }}
          {{ .Annotations.summary }}
          {{ .Annotations.description }}
          {{ end }}
    email_configs:
      - to: "${ALERT_TO}"
        send_resolved: true

The routing tree reads top-to-bottom: every alert starts at the root, and child routes with match filters can hand off before the default receiver is hit. inhibit_rules suppress warning alerts when a critical on the same host is firing — no point waking someone with "high CPU" if the host is down.

Step 5: Loki and Promtail Configuration

Create /opt/monitoring/configs/loki/loki-config.yml:

auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9096 log_level: info common: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory schema_config: configs: - from: 2024-01-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h limits_config: reject_old_samples: true reject_old_samples_max_age: 168h retention_period: 720h # 30 days max_query_series: 5000 max_query_parallelism: 32 ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 compactor: working_directory: /loki/compactor compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 delete_request_store: filesystem ruler: storage: type: local local: directory: /loki/rules rule_path: /loki/rules-tmp alertmanager_url: http://alertmanager:9093 ring: kvstore: store: inmemory enable_api: true

analytics: reporting_enabled: false

Retention is enforced by the compactor; the filesystem backend is perfectly fine up to a few hundred GB. Past that, flip the storage: stanza to s3: and point it at Backblaze B2, Wasabi, or AWS — see the scaling section at the end.

Create /opt/monitoring/configs/promtail/promtail-config.yml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: info
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
    batchwait: 1s
    batchsize: 1048576
scrape_configs:
  # -------- Host syslog + auth --------
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          host: monitoring-vps
          __path__: /var/log/{syslog,auth.log,kern.log}  # -------- Docker container logs --------
  - job_name: containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 10s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: container
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: logstream
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: stack
    pipeline_stages:
      - cri: {}
      - match:
          selector: '{service="nginx"}'
          stages:
            - regex:
                expression: '^(?P<remote>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<size>\d+)'
            - labels:
                method:
                status:

The pipeline stage for nginx is an example — drop it in for any service whose logs you want parsed into structured fields. Stay conservative: every label in Loki is a dimension, and dimensions are expensive.

Step 6: Grafana Provisioning

Grafana can be completely configured from files, so the container comes up with datasources wired and dashboards loaded — no click-ops needed.

Create /opt/monitoring/configs/grafana/provisioning/datasources/datasources.yml:

apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus:9090 isDefault: true jsonData: timeInterval: 15s httpMethod: POST - name: Loki type: loki access: proxy url: http://loki:3100 jsonData: maxLines: 1000 derivedFields: - name: TraceID matcherRegex: 'trace_id=(\w+)' url: '$${__value.raw}'

- name: Alertmanager type: alertmanager access: proxy url: http://alertmanager:9093 jsonData: implementation: prometheus

Create /opt/monitoring/configs/grafana/provisioning/dashboards/dashboards.yml:

apiVersion: 1

providers: - name: default orgId: 1 folder: '' type: file disableDeletion: false editable: true updateIntervalSeconds: 30 options: path: /var/lib/grafana/dashboards

Grafana will now load every JSON file in /opt/monitoring/configs/grafana/dashboards/ on start. Download a handful of community dashboards to get rolling:

cd /opt/monitoring/configs/grafana/dashboards
Node Exporter Full — ID 1860
curl -sSL "https://grafana.com/api/dashboards/1860/revisions/latest/download" \
  -o node-exporter-full.json
Docker cAdvisor — ID 19792
curl -sSL "https://grafana.com/api/dashboards/19792/revisions/latest/download" \
  -o docker-cadvisor.json
Loki logs — ID 13639
curl -sSL "https://grafana.com/api/dashboards/13639/revisions/latest/download" \
  -o loki-logs.json
Prometheus self-monitoring — ID 3662
curl -sSL "https://grafana.com/api/dashboards/3662/revisions/latest/download" \
  -o prometheus-stats.json

Provisioned dashboards reference data sources by UID which is unknown at render time, so we normalise to our provisioned names:

for f in *.json; do
  sed -i 's/"${DS_PROMETHEUS}"/"Prometheus"/g; s/"${DS_LOKI}"/"Loki"/g' "$f"
done

Create /opt/monitoring/configs/grafana/grafana.ini:

[server]
root_url = %(protocol)s://%(domain)s/
domain = grafana.example.com
enforce_domain = false
[users]
allow_sign_up = false
default_theme = dark
[auth]
disable_login_form = false
[auth.anonymous]
enabled = false
[security]
cookie_secure = true
cookie_samesite = lax
strict_transport_security = true
[analytics]
reporting_enabled = false
check_for_updates = true[unified_alerting]
enabled = false

We disable Grafana's unified alerting and let Alertmanager handle all alerting. Running two alerting systems leads to duplicate notifications.

Step 7: The Complete docker-compose.yml

Save this as /opt/monitoring/docker-compose.yml:

name: monitoring networks: monitoring: driver: bridge volumes: prometheus-data: alertmanager-data: grafana-data: loki-data: services: # ---------------------------------------- # Prometheus — metrics TSDB # ---------------------------------------- prometheus: image: prom/prometheus:v2.54.1 container_name: prometheus restart: unless-stopped command: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention.time=${PROMETHEUS_RETENTION} - --storage.tsdb.retention.size=${PROMETHEUS_RETENTION_SIZE} - --web.enable-lifecycle - --web.enable-admin-api - --web.external-url=http://prometheus:9090 volumes: - ./configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - ./configs/prometheus/rules:/etc/prometheus/rules:ro - prometheus-data:/prometheus ports: - "127.0.0.1:9090:9090" networks: - monitoring healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:9090/-/healthy"] interval: 30s timeout: 10s retries: 3 # ---------------------------------------- # Alertmanager — alert router # ---------------------------------------- alertmanager: image: prom/alertmanager:v0.27.0 container_name: alertmanager restart: unless-stopped command: - --config.file=/etc/alertmanager/alertmanager.yml - --storage.path=/alertmanager - --web.external-url=http://alertmanager:9093 environment: - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL} - SMTP_SMARTHOST=${SMTP_SMARTHOST} - SMTP_USER=${SMTP_USER} - SMTP_PASSWORD=${SMTP_PASSWORD} - ALERT_FROM=${ALERT_FROM} - ALERT_TO=${ALERT_TO} volumes: - ./configs/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro - alertmanager-data:/alertmanager ports: - "127.0.0.1:9093:9093" networks: - monitoring healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:9093/-/healthy"] interval: 30s timeout: 10s retries: 3 # ---------------------------------------- # Grafana — dashboards + explore # ---------------------------------------- grafana: image: grafana/grafana:11.2.0 container_name: grafana restart: unless-stopped depends_on: prometheus: condition: service_healthy loki: condition: service_healthy environment: - TZ=${TZ} - GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER} - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD} - GF_SERVER_ROOT_URL=${GF_SERVER_ROOT_URL} - GF_USERS_ALLOW_SIGN_UP=${GF_USERS_ALLOW_SIGN_UP} - GF_INSTALL_PLUGINS=grafana-piechart-panel volumes: - ./configs/grafana/grafana.ini:/etc/grafana/grafana.ini:ro - ./configs/grafana/provisioning:/etc/grafana/provisioning:ro - ./configs/grafana/dashboards:/var/lib/grafana/dashboards:ro - grafana-data:/var/lib/grafana ports: - "127.0.0.1:3000:3000" networks: - monitoring healthcheck: test: ["CMD-SHELL", "wget -qO- http://localhost:3000/api/health | grep -q ok"] interval: 30s timeout: 10s retries: 3 # ---------------------------------------- # Loki — log aggregation # ---------------------------------------- loki: image: grafana/loki:3.1.1 container_name: loki restart: unless-stopped command: -config.file=/etc/loki/loki-config.yml volumes: - ./configs/loki/loki-config.yml:/etc/loki/loki-config.yml:ro - loki-data:/loki ports: - "127.0.0.1:3100:3100" networks: - monitoring healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:3100/ready"] interval: 30s timeout: 10s retries: 5 # ---------------------------------------- # Promtail — log shipper # ---------------------------------------- promtail: image: grafana/promtail:3.1.1 container_name: promtail restart: unless-stopped depends_on: loki: condition: service_healthy command: -config.file=/etc/promtail/promtail-config.yml volumes: - ./configs/promtail/promtail-config.yml:/etc/promtail/promtail-config.yml:ro - /var/log:/var/log:ro - /var/lib/docker/containers:/var/lib/docker/containers:ro - /var/run/docker.sock:/var/run/docker.sock:ro networks: - monitoring # ---------------------------------------- # Node Exporter — host metrics # ---------------------------------------- node-exporter: image: prom/node-exporter:v1.8.2 container_name: node-exporter restart: unless-stopped command: - --path.procfs=/host/proc - --path.sysfs=/host/sys - --path.rootfs=/rootfs - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/) volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro pid: host networks: - monitoring healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:9100/metrics"] interval: 30s timeout: 10s retries: 3

# ---------------------------------------- # cAdvisor — container metrics # ---------------------------------------- cadvisor: image: gcr.io/cadvisor/cadvisor:v0.49.1 container_name: cadvisor restart: unless-stopped command: - --housekeeping_interval=30s - --docker_only=true volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:ro devices: - /dev/kmsg:/dev/kmsg privileged: true networks: - monitoring healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:8080/healthz"] interval: 30s timeout: 10s retries: 3

Key design decisions:

All HTTP ports bound to 127.0.0.1. Nothing reachable from the internet except through nginx.
Prometheus has a disk-size retention cap so runaway cardinality cannot blow out the VPS.
Loki and Prometheus have health checks that gate Grafana's startup — no UI errors on first load.
Named volumes for all state. Bind-mounted configs so edits round-trip with git.
Node Exporter runs host-PID so it can see all processes, not just containers.

Step 8: Launch the Stack

cd /opt/monitoring
docker compose pull
docker compose up -d
docker compose ps

All seven services should reach healthy within 60-90 seconds. Tail the logs to watch for config errors:

docker compose logs -f --tail=100

Quick health checks:

curl -s http://127.0.0.1:9090/-/healthy           # Prometheus
curl -s http://127.0.0.1:9093/-/healthy           # Alertmanager
curl -s http://127.0.0.1:3000/api/health | jq .   # Grafana
curl -s http://127.0.0.1:3100/ready               # Loki

Confirm Prometheus is scraping every target:

curl -s http://127.0.0.1:9090/api/v1/targets | \
  jq '.data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health}'

Every target should report "health": "up". If anything says down, click through to Prometheus's Targets page (http://127.0.0.1:9090/targets via SSH tunnel) for the exact scrape error.

Step 9: Import Dashboards and Validate

Grafana should already have Node Exporter Full, cAdvisor, Loki, and Prometheus dashboards loaded via provisioning. Tunnel in:

ssh -L 3000:127.0.0.1:3000 youruser@yourserver

Visit http://localhost:3000, sign in with the admin credentials from .env, and check:

Datasources (Connections -> Data sources). Prometheus, Loki, and Alertmanager all green. Hit "Save & test" on each to confirm.

Dashboards (Dashboards). The four pre-loaded dashboards should appear. Open "Node Exporter Full" — it should render immediately with CPU, memory, disk, and network panels populated.

Explore (Explore -> Prometheus). Try: rate(node_cpu_seconds_total[5m]) or 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100).

Explore -> Loki. Try: {container="grafana"} or {container=~".+"} |= "error".

To import additional community dashboards on the fly, use Dashboards -> New -> Import and paste an ID from grafana.com/dashboards. Popular picks:

Dashboard	ID	What it shows
Node Exporter Full	1860	Everything about a Linux host
Docker cAdvisor	19792	Per-container CPU, memory, I/O
Loki Logs / App	13639	Log volume per label
Prometheus 2.0 Stats	3662	Prometheus internal health
Alertmanager	9578	Silenced/active alerts, notify success
Nginx	12708	Requests, status codes, latency
Postgres	9628	Connections, transactions, locks
Redis	763	Commands, memory, keyspace

Imported dashboards land in the default folder. Pick the "Prometheus" datasource when Grafana prompts during import.

Step 10: Alert Rules and LogQL Examples

Test an alert end-to-end. Force an alert by editing the HostHighCpu expression to > 0 and reloading Prometheus:

curl -X POST http://127.0.0.1:9090/-/reload

Within 30 seconds the Prometheus Alerts page should show HostHighCpu firing; within 60 seconds it should land in Alertmanager and a Slack message should hit #alerts. Revert the expression and reload again.

LogQL cheat sheet. Useful queries to save as starred Explore queries:

# Every error line across the whole stack, last 15 minutes
{container=~".+"} |= "error" != "error_rate"
Nginx 5xx spikes (after you enabled the nginx pipeline stage)
sum by (status) (rate({service="nginx"} | status=~"5.."[5m]))
Loki ingest rate — spot traffic surges
sum(rate({container=~".+"}[1m]))
Container restart explanations — last 10 minutes of logs for a restarting container
{container="api"} | json | level=~"error|fatal"
Correlate errors with trace IDs (requires the derivedFields you set in datasource)
{container="api"} |= "error" | regexp "trace_id=(?P<trace>\\w+)"

PromQL for SLO-style alerts. A production-ready template for an availability SLO on an HTTP service exposing standard http_requests_total{status} metrics:

- alert: AvailabilityBudgetBurn
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
    ) > (1 - 0.995) * 14.4
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Availability budget burn rate is high"
    description: "5xx rate has been over 14.4x the SLO target for 5 minutes."

A 14.4x burn rate over a 30-day 99.5% SLO exhausts 2% of the monthly error budget in one hour — the multi-window multi-burn-rate pattern from Google's SRE workbook. Drop PROMETHEUS_TARGET_SLO variations into rules/ as your services harden.

Step 11: nginx Reverse Proxy with SSL

Install nginx and certbot on the host:

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/conf.d/monitoring.conf:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 80;
    server_name grafana.example.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name grafana.example.com;
    ssl_certificate     /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_session_cache   shared:SSL:10m;
    # Hard security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-Frame-Options SAMEORIGIN always;
    add_header Referrer-Policy strict-origin-when-cross-origin always;
    client_max_body_size 20M;
    location / {
        proxy_pass         http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        # Grafana Live uses WebSockets for streaming
        proxy_set_header   Upgrade    $http_upgrade;
        proxy_set_header   Connection $connection_upgrade;        proxy_buffering    off;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Deliberately no server blocks for Prometheus, Loki, or Alertmanager. Those services stay internal; you reach them from Grafana's Explore and Alerting pages. If you absolutely must expose one — say a Prometheus URL for a remote CI job to push custom metrics to — add basic auth and an IP allowlist.

Provision SSL:

sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx -d grafana.example.com \
  --agree-tos -m [email protected] --redirect

Certbot installs a systemd timer for auto-renewal — confirm with sudo systemctl list-timers | grep certbot.

Visit https://grafana.example.com. Login, green padlock, HSTS header — you are live.

Step 12: Backups

Three things need backing up: Prometheus TSDB (metrics history), Grafana DB (users, dashboards, API keys, alert state), and Loki chunks (your logs). Config files live in /opt/monitoring/configs/ and should already be in git.

Save this as /opt/monitoring/scripts/backup.sh:

#!/usr/bin/env bash set -euo pipefail BACKUP_ROOT=/opt/monitoring/backups STAMP=$(date +%Y%m%d-%H%M%S) DEST="$BACKUP_ROOT/$STAMP" mkdir -p "$DEST" cd /opt/monitoring 1. Prometheus — snapshot via the admin API (no restart) echo "[*] Snapshotting Prometheus TSDB..." SNAP=$(curl -sS -X POST http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot | jq -r .data.name) docker run --rm \ -v monitoring_prometheus-data:/data:ro \ -v "$DEST":/backup \ alpine tar czf /backup/prometheus-snapshot.tar.gz -C "/data/snapshots/$SNAP" . 2. Grafana — SQLite DB copy echo "[*] Backing up Grafana..." docker run --rm \ -v monitoring_grafana-data:/data:ro \ -v "$DEST":/backup \ alpine tar czf /backup/grafana-data.tar.gz -C /data . 3. Loki — chunk dir (compactor handles consistency) echo "[*] Backing up Loki chunks..." docker run --rm \ -v monitoring_loki-data:/data:ro \ -v "$DEST":/backup \ alpine tar czf /backup/loki-data.tar.gz -C /data . 4. Alertmanager silence state docker run --rm \ -v monitoring_alertmanager-data:/data:ro \ -v "$DEST":/backup \ alpine tar czf /backup/alertmanager-data.tar.gz -C /data . 5. Config files tar czf "$DEST/configs.tar.gz" .env configs docker-compose.yml 6. Rotate — keep 14 daily backups locally find "$BACKUP_ROOT" -maxdepth 1 -type d -mtime +14 -exec rm -rf {} \;

echo "[*] Backup complete: $DEST" du -sh "$DEST"

Make executable and schedule:

chmod +x /opt/monitoring/scripts/backup.sh
sudo crontab -e
0 3   * /opt/monitoring/scripts/backup.sh >> /var/log/monitoring-backup.log 2>&1

Off-site with rclone (recommended — local backups do not help if the VPS dies):

sudo apt install -y rclone
rclone config   # set up Backblaze B2 or Wasabi
Append to backup.sh:
rclone sync "$BACKUP_ROOT" b2:my-monitoring-backups/ --transfers 4

Restore drill. Once a quarter, spin up a fresh VPS, rsync the latest backup, restore each volume from its tarball, docker compose up -d, and confirm Grafana loads with all dashboards intact. A backup you have never restored from is not a backup.

Scaling Considerations

Vertical first. The recommended tier handles 50 hosts and a few hundred GB of logs comfortably. Before anything clever, resize the VPS — CloudCore tiers let you bump vCPU and RAM in place.

Prometheus: Thanos or Mimir. Once a single Prometheus is under pressure or you need long-term (years) retention and global querying, the pattern is to keep your existing Prometheus and deploy Thanos or Grafana Mimir alongside it. Thanos Sidecar uploads TSDB blocks to S3 every 2 hours; Thanos Query federates multiple Prometheus instances behind a single endpoint. Grafana Mimir is the alternative from Grafana Labs — same idea, different operator experience. Either way you stop worrying about local disk and scale horizontally.

Loki: S3 backend. Local filesystem is fine up to a few hundred GB. Past that, flip the storage: stanza in loki-config.yml to S3:

common:
  storage:
    s3:
      endpoint: s3.us-west-004.backblazeb2.com
      region: us-west-004
      bucket: monitoring-logs
      access_key_id: "${B2_KEY_ID}"
      secret_access_key: "${B2_APP_KEY}"
      s3forcepathstyle: true

Backblaze B2 S3-compatible storage runs roughly $6/TB/month and scales to petabytes. Combined with Loki's label-indexed architecture you can retain a year of logs for less than a week of Datadog.

Sharding Promtail. One Promtail per host. Do not run one central Promtail scraping remote Docker hosts; ship logs locally and push over the network. This keeps Loki's ingress protocol simple and avoids a single point of failure.

Federating Alertmanager. For multi-region setups, run one Alertmanager per region and cluster them with --cluster.peer= flags. Deduplication happens across the cluster so the same alert does not wake three people.

Cardinality hygiene. The one thing that will torpedo a Prometheus install is label explosion — putting a unique ID (request ID, email, UUID) into a label. Prometheus stores one TSDB series per unique label set; a million request IDs creates a million series and kills the server. Rules of thumb: labels should be bounded and human-enumerable (status code, HTTP method, tenant ID if you have fewer than a few thousand tenants). Put high-cardinality values in log lines (Loki) or traces (Tempo), not in metric labels.

Troubleshooting

Prometheus "context deadline exceeded" on scrape. The target is slow, firewalled, or down. Check from the Prometheus container: docker compose exec prometheus wget -qO- http://target:9100/metrics. For remote targets behind a cloud firewall, open the port only to the Prometheus VPS IP or use a WireGuard / Tailscale overlay.

Node Exporter showing no disk metrics. rootfs and filesystem-related volumes are not mounted correctly. The compose file sets /:/rootfs:ro and the --path.rootfs=/rootfs flag — both are required. Recreate with docker compose up -d --force-recreate node-exporter.

Loki returns 400 "entry too far behind". Promtail has caught up on a log file older than reject_old_samples_max_age (168h). Either increase the limit in loki-config.yml or stop shipping archival files. Fresh lines will still ingest fine.

Grafana "datasource not found" on a provisioned dashboard. The dashboard JSON references a datasource UID that does not match what Grafana generated for your provisioned datasource. Fix: replace the UID reference in the dashboard JSON with the datasource name ("Prometheus" / "Loki") and redeploy. The sed one-liner in Step 6 does this for the canonical community dashboards.

High cardinality warnings in Prometheus logs. WARN ts=... msg="target has too many samples" or exemplar storage is full. Run topk(20, count by (__name__)({__name__=~".+"})) in the expression browser to find the bloated metrics. Usually the culprit is an app label like customer_email. Drop the offending labels via a metric_relabel_configs stanza in the scrape job:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'myapp_requests_total'
    action: labeldrop
    regex: 'customer_email'

Storage bloat — Prometheus disk filling up. --storage.tsdb.retention.size=20GB is the hard cap. Prometheus rotates blocks when it hits the cap but needs free space equal to the biggest block (~2h of data) to compact. Leave 25% headroom. If logs are the culprit, check Loki's compactor: docker compose logs loki | grep compactor.

Alerts firing but nothing in Slack. Three-step check: (1) Prometheus Alerts page shows the alert as firing; (2) Alertmanager UI (tunnel :9093) shows the alert in the active list; (3) the amtool CLI can reach the Slack webhook. If (2) is missing, the alerting: stanza in prometheus.yml is wrong. If (3) fails, test manually: curl -X POST -H 'Content-type: application/json' --data '{"text":"test"}' $SLACK_WEBHOOK_URL.

Grafana auth.proxy loop or redirect issues. root_url in grafana.ini and GF_SERVER_ROOT_URL in the compose file must match exactly, including scheme and trailing path. Behind nginx the simplest posture is root_url=https://grafana.example.com/ with enforce_domain=false.

Grafana datasource "authentication failed". Prometheus and Loki in this stack do not require auth — the Grafana datasource should have auth set to "none". If you later add basic auth in front of Prometheus (say via nginx), set the matching credentials in the datasource yaml, not the URL.

cAdvisor missing some containers. cAdvisor needs privileged: true and /sys mounted. On cgroup v2 hosts (Ubuntu 24.04 default) older cAdvisor versions misreport some metrics — pin to v0.49.1 or later.

FAQ

How does this compare to Datadog? Datadog is a turn-key SaaS at $31-70/host/month. The self-hosted stack here runs ~$40/month total regardless of host count, and owns its data. You trade convenience for control and cost. Most teams past 3-5 hosts come out ahead by four figures a year.

How does this compare to ELK (Elasticsearch + Logstash + Kibana)? ELK indexes the full text of every log line, which makes arbitrary search fast but means storage scales close to 1:1 with ingest. Loki indexes only labels, compresses chunks hard, and scans text on demand — roughly a 10x storage cost improvement for the same log volume. ELK is still the right choice if you need complex full-text queries with millisecond latency; Loki is the right choice if you want cheap, bounded log retention.

How does this compare to SigNoz? SigNoz is an all-in-one metrics + logs + traces tool built on ClickHouse, with a single UI. It is excellent and worth evaluating. The Prometheus stack is the industry standard — every exporter, every dashboard, every on-call engineer's muscle memory assumes it. The migration path from this stack to SigNoz is harder than the reverse, so start here.

How long can I retain metrics and logs? With the recommended 4 vCPU / 8 GB / 80 GB VPS and default settings: 30 days of metrics (cap 20 GB) and 30 days of logs (cap ~40 GB). For 90+ days of metrics add Thanos with S3; for 90+ days of logs flip Loki to the S3 backend.

How do I avoid alert fatigue? Three habits. (1) Every alert must be actionable — if the response is "ignore it" or "wait and see", the rule is bad. (2) Use for: liberally; a 10-minute for on "high CPU" eliminates most flapping. (3) Use Alertmanager inhibit_rules (we have one in this guide) so a critical suppresses lesser alerts on the same instance. Audit your alerts quarterly: if a given alert has not fired or has been silenced repeatedly, delete it.

Can I monitor remote hosts across the internet? Yes, but do not expose Node Exporter on a public IP. Put every monitored host on a mesh VPN (Tailscale, WireGuard, Nebula). Prometheus scrapes the private addresses over the overlay. This is how every production observability deployment is laid out.

How do I add a new service to monitor? Three ways depending on the service. (1) If it exposes Prometheus metrics natively (most modern services do), add a scrape_config and reload Prometheus. (2) If it does not, install the appropriate Prometheus exporter (postgres_exporter, redis_exporter, blackbox_exporter for HTTP / TCP / ICMP checks). (3) For logs, add a scrape_config to Promtail with the right labels and pipeline stages. The Grafana dashboard gallery usually has something prebuilt — search by the exporter name.

Can I replace Alertmanager with Grafana's unified alerting? You can, but do not. Grafana alerting duplicates rules across data sources and makes it harder to silence alerts during maintenance. The canonical production path is Prometheus evaluates rules -> Alertmanager routes. Leave unified_alerting.enabled=false as set in this guide.

How do I upgrade? docker compose pull && docker compose up -d. Pin the major/minor version (as we do with prom/prometheus:v2.54.1) so a point release does not silently introduce a breaking config change. Before any major bump, run the backup script first and check the release notes for deprecated flags.

Can I run this alongside an existing ELK or Datadog agent? Absolutely. Many teams run the Prometheus stack for metrics and alerting while keeping ELK for log search. Grafana's Elasticsearch datasource connects to your existing cluster and shows both types of data in one dashboard.

Does Grafana have multi-tenancy? Grafana OSS supports organisations, folders with per-folder permissions, and per-user API keys. For a hosting company giving each tenant their own dashboards, folders with RBAC plus Auth Proxy headers is the pattern most teams use. Full multi-tenancy with isolated Prometheus and Loki is a Grafana Enterprise feature (or a self-operated Mimir / Loki cluster with the X-Scope-OrgID header wired up).

Next Steps

You now own a production-capable monitoring stack. Sensible follow-ups:

How to Install Grafana on Ubuntu 24.04 — deeper dive into Grafana's configuration surface, OIDC, and plugin management.
How to Install Prometheus on Ubuntu 24.04 — native (non-Docker) Prometheus installation, remote-write, and Thanos integration.
How to Install Loki on Ubuntu 24.04 — Loki-specific deep dive on chunk formats, LogQL, and S3 backends.
How to Install Uptime Kuma on Ubuntu 24.04 — pair with Uptime Kuma for blackbox / external reachability checks.
How to Secure SSH on Ubuntu — harden the monitoring host itself.
How to Configure rclone for Off-Site Backups — automate the off-site copy of your monitoring backups.

If you run into anything this guide does not cover, our support team is happy to help — open a ticket from the vps-server.host dashboard and attach docker compose logs --tail=500 > logs.txt.

Want the stack running without the wrench time? Our CloudCore Business plan at EUR 35.99/month is the exact tier sized for this build — 4 vCPU, 8 GB RAM, 80 GB NVMe, Ubuntu 24.04 preinstalled, Docker ready on boot. Launch a monitoring VPS now and follow this guide start-to-finish in under an hour.

Was this article helpful?

← Back to Install Guides Browse all categories →

Still have questions?

Contact Support →Submit a Ticket

How to Build a Self-Hosted Monitoring Stack (Prometheus + Grafana + Loki) on Ubuntu 24.04

Table of Contents

Architecture Overview

Why Self-Host Observability?

Cost Comparison: Self-Hosted vs Datadog

Prerequisites and VPS Sizing

The Components Explained

Step 1: Prepare the Server

Step 2: Directory Layout and Environment File

--- Grafana ---

--- Alert notifiers ---

--- Retention ---

Step 3: Prometheus Configuration

Step 4: Alertmanager Configuration

Step 5: Loki and Promtail Configuration

Step 6: Grafana Provisioning

Node Exporter Full — ID 1860

Docker cAdvisor — ID 19792

Loki logs — ID 13639

Prometheus self-monitoring — ID 3662

Step 7: The Complete docker-compose.yml

Step 8: Launch the Stack

Step 9: Import Dashboards and Validate

Step 10: Alert Rules and LogQL Examples

Nginx 5xx spikes (after you enabled the nginx pipeline stage)

Loki ingest rate — spot traffic surges

Container restart explanations — last 10 minutes of logs for a restarting container

Correlate errors with trace IDs (requires the derivedFields you set in datasource)

Step 11: nginx Reverse Proxy with SSL

Step 12: Backups

1. Prometheus — snapshot via the admin API (no restart)

2. Grafana — SQLite DB copy

3. Loki — chunk dir (compactor handles consistency)

4. Alertmanager silence state

5. Config files

6. Rotate — keep 14 daily backups locally

0 3 * /opt/monitoring/scripts/backup.sh >> /var/log/monitoring-backup.log 2>&1

Append to backup.sh:

rclone sync "$BACKUP_ROOT" b2:my-monitoring-backups/ --transfers 4

Scaling Considerations

Troubleshooting

FAQ

Next Steps

How to Build a Self-Hosted Monitoring Stack (Prometheus + Grafana + Loki) on Ubuntu 24.04

Table of Contents

Architecture Overview

Why Self-Host Observability?

Cost Comparison: Self-Hosted vs Datadog

Prerequisites and VPS Sizing

The Components Explained

Step 1: Prepare the Server

Step 2: Directory Layout and Environment File

--- Grafana ---

--- Alert notifiers ---

--- Retention ---

Step 3: Prometheus Configuration

Step 4: Alertmanager Configuration

Step 5: Loki and Promtail Configuration

Step 6: Grafana Provisioning

Node Exporter Full — ID 1860

Docker cAdvisor — ID 19792

Loki logs — ID 13639

Prometheus self-monitoring — ID 3662

Step 7: The Complete docker-compose.yml

Step 8: Launch the Stack

Step 9: Import Dashboards and Validate

Step 10: Alert Rules and LogQL Examples

Nginx 5xx spikes (after you enabled the nginx pipeline stage)

Loki ingest rate — spot traffic surges

Container restart explanations — last 10 minutes of logs for a restarting container

Correlate errors with trace IDs (requires the derivedFields you set in datasource)

Step 11: nginx Reverse Proxy with SSL

Step 12: Backups

1. Prometheus — snapshot via the admin API (no restart)

2. Grafana — SQLite DB copy

3. Loki — chunk dir (compactor handles consistency)

4. Alertmanager silence state

5. Config files

6. Rotate — keep 14 daily backups locally

0 3 * /opt/monitoring/scripts/backup.sh >> /var/log/monitoring-backup.log 2>&1

`0 3 * /opt/monitoring/scripts/backup.sh >> /var/log/monitoring-backup.log 2>&1`

`rclone sync "$BACKUP_ROOT" b2:my-monitoring-backups/ --transfers 4`

`0 3 * /opt/monitoring/scripts/backup.sh >> /var/log/monitoring-backup.log 2>&1`

`rclone sync "$BACKUP_ROOT" b2:my-monitoring-backups/ --transfers 4`