How to Install Loki on Ubuntu 24.04 VPS — Self-Hosted Log Aggregation
Centralized logging is one of the three pillars of observability, but most solutions are either painfully expensive (Datadog, Splunk) or operationally heavy (Elasticsearch, Graylog). Grafana Loki takes a different approach: index only metadata labels, store log content in cheap object storage, and query with a Prometheus-like language called LogQL. The result is a log aggregation system that costs a fraction of Elasticsearch-based stacks while scaling to terabytes of logs per day.
This guide walks you through installing Loki on an Ubuntu 24.04 LTS VPS, from the first SSH connection to a production deployment with S3-backed storage, Promtail agents forwarding logs from multiple servers, retention policies, and a Grafana dashboard visualizing everything. By the end you will have a working log pipeline you can point any application, container, or syslog source at.
Looking for a managed Grafana + Loki stack? Our Professional VPS plan provides the right balance of CPU, RAM, and NVMe storage for a single-node Loki deployment serving 10-50 GB of logs per day.
Table of Contents
What is Loki?
Grafana Loki is an open-source, horizontally scalable log aggregation system inspired by Prometheus. Where Prometheus indexes metrics with labels, Loki indexes log streams with labels — but crucially, it does not index the log content itself. Instead, raw log lines are compressed and pushed to an object store (S3, GCS, Azure Blob, or self-hosted MinIO), while a small index of labels is kept fast-queryable.
This design sounds counterintuitive at first — how can you search logs if you do not index the content? The answer is LogQL, Loki's query language, which uses labels to narrow down the relevant streams first and then greps through the compressed content on the fly. Because labels filter the dataset down to a few gigabytes before the full-text scan happens, queries stay fast without the storage and memory overhead of an inverted index.
Loki is developed by Grafana Labs, the same team behind Grafana and the Grafana Tempo tracing backend, and is released under the AGPL-3.0 license. The three systems are designed to work together as a cohesive observability stack: Loki for logs, Prometheus for metrics, and Tempo for traces, all queried through a single Grafana UI.
The ecosystem includes several companion tools. Promtail is the official log-shipping agent, responsible for discovering log files on a host, adding labels, and pushing them to Loki. Grafana Agent and Vector are alternative collectors. For Kubernetes, the Loki Helm chart ships with Promtail as a DaemonSet that auto-discovers pod logs. And the logcli CLI tool lets you query Loki from the terminal, which is useful for scripting and alerts.
Why Self-Host Loki Instead of Elasticsearch, Graylog, or CloudWatch?
Log aggregation is one of those problems where the default choice — send everything to a hosted SaaS — quickly becomes the most expensive line item on your infrastructure bill. Here is how Loki compares to the main alternatives.
Loki vs Elasticsearch (ELK stack). Elasticsearch is the historical standard for log search, and it is extremely powerful — full-text indexing, fuzzy queries, aggregations, vector search. But that power comes at a cost: Elasticsearch typically needs 3-10x more RAM and disk than Loki for the same log volume, because it maintains a full inverted index. A 100 GB/day log pipeline on ELK often requires 32-64 GB RAM and NVMe SSDs; the same volume on Loki runs comfortably on 12 GB RAM and cheap S3 storage. Loki also has a much simpler operational model — a single binary, no JVM tuning, no shard rebalancing, no split brains.
Loki vs Graylog. Graylog is a polished ELK-based product with built-in dashboards, alerting, and role management. It is a great choice if you need GELF input, complex stream routing, and a turnkey UI. But Graylog carries all of Elasticsearch's operational weight plus a MongoDB dependency for configuration. Loki is the better fit when you already use Grafana for metrics and want a unified observability UI without running a separate logging product.
Loki vs AWS CloudWatch Logs. CloudWatch is convenient if your workload lives entirely in AWS, but the pricing scales brutally: $0.50 per GB ingested, plus storage, plus $0.005 per query-GB scanned. A team ingesting 50 GB/day pays roughly $750/month on ingestion alone before a single query runs. The equivalent self-hosted Loki setup on a Professional VPS with S3-backed storage costs under EUR 30/month, including bandwidth.
Cost Comparison: Loki vs Alternatives at 50 GB/day Ingestion
| Solution | Monthly Cost | Index Size | Query Language | Ops Complexity |
|---|---|---|---|---|
| Self-hosted Loki (VPS + S3) | ~EUR 29/mo | Tiny (labels only) | LogQL | Low (single binary) |
| Self-hosted Elasticsearch | ~EUR 79/mo (bigger VPS) | Full inverted index | Lucene/KQL | High (JVM, shards) |
| Self-hosted Graylog | ~EUR 89/mo | Full inverted index | GL Search | Medium (ES + Mongo) |
| AWS CloudWatch Logs | ~$750/mo | Managed | Logs Insights | Zero (but locked-in) |
| Datadog Logs | ~$1,200/mo | Managed | Datadog query | Zero (but locked-in) |
Prerequisites
Before you begin, make sure you have:
- A VPS running Ubuntu 24.04 LTS with root or sudo access
- SSH access to your server
- At least 8 GB of RAM (12 GB recommended for production workloads)
- At least 50 GB of local NVMe storage for the write-ahead log, BoltDB index cache, and chunk buffers (200 GB recommended)
- An S3-compatible object store — AWS S3, Backblaze B2, Wasabi, or a self-hosted MinIO instance
- An existing Grafana installation (on the same or a different server) for querying logs
Recommended Plan: CloudCore Professional>
For a single-node Loki deployment ingesting 10-50 GB of logs per day, we recommend the Professional VPS plan:>
- 6 vCPU cores
- 12 GB RAM
- 200 GB NVMe SSD
- Unmetered bandwidth
- EUR 19.99/month>
This gives you enough headroom for Loki's ingester, BoltDB-shipper cache, and Promtail sidecar traffic, while leaving resources for Grafana on the same node. For larger ingestion rates (100+ GB/day) or high availability, move to the Simple Scalable mode across multiple VPS instances — our CloudCore Business plan (8 vCPU / 24 GB RAM) at EUR 29.99/month is a good fit for write nodes.
CloudCore Pricing Tiers for Loki
| Plan | Specs | Ingestion Capacity | Monthly Price |
|---|---|---|---|
| CloudCore Starter | 2 vCPU / 4 GB / 50 GB | Up to 5 GB/day | EUR 7.99 |
| CloudCore Professional | 6 vCPU / 12 GB / 200 GB | 5-50 GB/day | EUR 19.99 |
| CloudCore Business | 8 vCPU / 24 GB / 400 GB | 50-150 GB/day | EUR 29.99 |
ssh root@your-server-ipStep 1: Update System Packages and Create the Loki User
Start by updating your package index and installing a few dependencies we will use later (unzip, curl, wget).
sudo apt update && sudo apt upgrade -y
sudo apt install -y unzip curl wget gnupg ca-certificatesCreate a dedicated system user for running Loki. Running services under an unprivileged user is a basic security hygiene step — if the Loki process is ever compromised, the attacker cannot modify system binaries or read other users' files.
sudo groupadd --system loki
sudo useradd --system --no-create-home --shell /usr/sbin/nologin --gid loki lokiCreate the directories Loki will use for configuration, data, and logs:
sudo mkdir -p /etc/loki /var/lib/loki /var/log/loki
sudo chown -R loki:loki /var/lib/loki /var/log/loki
sudo chmod 750 /etc/loki /var/lib/lokiStep 2: Download and Install the Loki Binary
Grafana Labs publishes pre-compiled binaries for every Loki release on GitHub. We will pin to a specific version to avoid surprise upgrades.
Download the latest stable release (at the time of writing, 3.3.2 — check the releases page for the newest version):
cd /tmp
LOKI_VERSION="3.3.2"
wget "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-amd64.zip"
unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki
sudo chmod +x /usr/local/bin/loki
sudo chown root:root /usr/local/bin/lokiVerify the install:
loki --versionExpected output:
loki, version 3.3.2 (branch: HEAD, revision: abc1234)
build user: root@buildkitsandbox
build date: 2026-03-04
go version: go1.22.9
platform: linux/amd64While we are at it, also install logcli, the Loki query tool. It is indispensable for debugging and scripting alerts.
cd /tmp
wget "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/logcli-linux-amd64.zip"
unzip logcli-linux-amd64.zip
sudo mv logcli-linux-amd64 /usr/local/bin/logcli
sudo chmod +x /usr/local/bin/logcliStep 3: Choose a Deployment Mode — Monolithic vs Simple Scalable
Loki can run in three architectural modes. Picking the right one upfront saves a migration later.
Monolithic mode runs all Loki components (distributor, ingester, querier, query-frontend, compactor) inside a single process. This is the simplest setup and what we will use for this guide. It scales vertically to roughly 100 GB/day of ingested logs on a well-provisioned VPS. Single-node monolithic is ideal for small teams, staging environments, and side projects where operational simplicity matters more than horizontal scale.
Simple Scalable mode splits Loki into two roles: read (querier + query-frontend) and write (distributor + ingester + compactor). Each role runs as its own process, and you can scale them independently — more write nodes during peak ingestion, more read nodes during dashboard-heavy hours. This is the recommended mode for production deployments ingesting 100 GB - 1 TB per day. It adds some operational complexity (a load balancer, shared object store, memberlist gossip) but remains much simpler than the fully microservices mode.
Microservices mode breaks every component into its own deployment. This is what Grafana Cloud uses internally, and it scales to petabytes per day. For almost everyone else, it is operational overkill.
Decision Table
| Ingestion Volume | Team Size | Recommended Mode | CloudCore Plan |
|---|---|---|---|
| < 10 GB/day | 1-5 engineers | Monolithic on a single VPS | Standard |
| 10-100 GB/day | 5-50 engineers | Monolithic | Professional |
| 100 GB - 1 TB/day | 50+ engineers | Simple Scalable (2 write, 2 read) | Business x 4 |
| 1 TB+/day | Platform team | Microservices / Grafana Cloud | Enterprise cluster |
Step 4: Configure loki.yaml
Loki's behavior is controlled by a single YAML file. Create /etc/loki/loki.yaml with the following configuration:
sudo tee /etc/loki/loki.yaml > /dev/null <<'EOF' auth_enabled: falseserver: http_listen_port: 3100 grpc_listen_port: 9096 log_level: info
common: instance_addr: 127.0.0.1 path_prefix: /var/lib/loki replication_factor: 1 ring: kvstore: store: inmemory
ingester: wal: enabled: true dir: /var/lib/loki/wal lifecycler: address: 127.0.0.1 ring: kvstore: store: inmemory replication_factor: 1 final_sleep: 0s chunk_idle_period: 1h max_chunk_age: 1h chunk_target_size: 1572864 chunk_retain_period: 30s
schema_config: configs: - from: 2024-01-01 store: tsdb object_store: s3 schema: v13 index: prefix: loki_index_ period: 24h
storage_config: tsdb_shipper: active_index_directory: /var/lib/loki/tsdb-index cache_location: /var/lib/loki/tsdb-cache aws: s3: s3://AKIAEXAMPLE:[email protected]/loki-chunks s3forcepathstyle: true
limits_config: reject_old_samples: true reject_old_samples_max_age: 168h retention_period: 744h max_query_series: 10000 max_query_parallelism: 32 ingestion_rate_mb: 16 ingestion_burst_size_mb: 32
compactor: working_directory: /var/lib/loki/compactor compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 delete_request_store: s3
ruler: storage: type: local local: directory: /etc/loki/rules rule_path: /var/lib/loki/rules-temp alertmanager_url: http://localhost:9093 ring: kvstore: store: inmemory enable_api: true
analytics: reporting_enabled: false EOF
sudo chown loki:loki /etc/loki/loki.yaml sudo chmod 640 /etc/loki/loki.yaml
Let's break down the key sections:
auth_enabled: false— Disables multi-tenancy. For a single-team deployment this is fine; all logs share tenant IDfake. Enable this and put Loki behind an auth proxy (Nginx + basic auth, or OAuth2 Proxy) if you plan to host logs for multiple tenants.server— HTTP on3100is the query and push endpoint; gRPC on9096is used for internal component communication. Keep these bound to localhost or your private network — do not expose them directly to the internet.ingester.wal— The write-ahead log protects against data loss if Loki crashes mid-flush. Keep WAL on local NVMe.schema_config— Tells Loki which storage schema to use starting from a given date.tsdbis the current recommended index format (replaces the olderboltdb-shipper) and ships with Loki 3.x. Thefromdate must be in the past, and you must never change or remove an existing schema period — add a new one if you need to migrate.storage_config.aws— The S3 connection string. We will fill in real credentials in Step 5.limits_config.retention_period: 744h— 31 days of retention. Loki only deletes logs older than this ifcompactor.retention_enabledis true.compactor— Compacts index files and enforces retention. Without the compactor, logs accumulate forever.
sudo -u loki /usr/local/bin/loki -config.file=/etc/loki/loki.yaml -verify-configExpected output:
config is validStep 5: Configure S3-Compatible Object Storage
The S3 connection string in loki.yaml is the most sensitive part of the config. Put real credentials there, not placeholders.
Option A: AWS S3
Create a bucket (via the AWS console or CLI) named loki-chunks-yourcompany. Create an IAM user with a policy that grants s3:ListBucket, s3:GetObject, s3:PutObject, and s3:DeleteObject only on that bucket.
Update the storage block:
storage_config:
tsdb_shipper:
active_index_directory: /var/lib/loki/tsdb-index
cache_location: /var/lib/loki/tsdb-cache
aws:
s3: s3://AKIAYOURKEY:[email protected]/loki-chunks-yourcompany
s3forcepathstyle: falseOption B: Self-Hosted MinIO
If you want full data sovereignty, run MinIO on another VPS and point Loki at it. Create a bucket called loki-chunks and an access key:
storage_config:
aws:
s3: s3://minioaccess:[email protected]:9000/loki-chunks
s3forcepathstyle: true
insecure: trueThe s3forcepathstyle: true flag is required for MinIO, Ceph, and most non-AWS S3 implementations — they use path-style addressing (host/bucket/object) rather than virtual-host-style (bucket.host/object).
Option C: Backblaze B2 or Wasabi
These are drop-in replacements for S3 at a fraction of the cost. Example for Backblaze:
storage_config:
aws:
s3: s3://B2KEYID:[email protected]/loki-chunks
s3forcepathstyle: trueSecuring Credentials
Hard-coding credentials in loki.yaml works but is not ideal. For production, prefer environment variables:
aws:
s3: s3://${S3_ACCESS_KEY}:${S3_SECRET_KEY}@s3.eu-central-1.amazonaws.com/loki-chunksAnd pass them via the systemd unit (see next step).
Step 6: Create the systemd Service
Create /etc/systemd/system/loki.service:
sudo tee /etc/systemd/system/loki.service > /dev/null <<'EOF' [Unit] Description=Grafana Loki log aggregation service Documentation=https://grafana.com/docs/loki/latest/ Wants=network-online.target After=network-online.target[Service] Type=simple User=loki Group=loki ExecStart=/usr/local/bin/loki \ -config.file=/etc/loki/loki.yaml \ -config.expand-env=true Restart=on-failure RestartSec=10 LimitNOFILE=65536 EnvironmentFile=-/etc/loki/loki.env
Hardening
NoNewPrivileges=true ProtectSystem=full ProtectHome=true PrivateTmp=true ReadWritePaths=/var/lib/loki /var/log/loki
[Install] WantedBy=multi-user.target EOF
Create the environment file for secrets:
sudo tee /etc/loki/loki.env > /dev/null <<'EOF'
S3_ACCESS_KEY=AKIAYOURKEY
S3_SECRET_KEY=YourSecretKey
EOF
sudo chown loki:loki /etc/loki/loki.env
sudo chmod 600 /etc/loki/loki.envReload systemd, enable, and start Loki:
sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki
sudo systemctl status lokiExpected output:
● loki.service - Grafana Loki log aggregation service
Loaded: loaded (/etc/systemd/system/loki.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-04-16 10:00:00 UTC; 5s ago
Main PID: 1234 (loki)
Tasks: 12
Memory: 128.4M
CGroup: /system.slice/loki.service
└─1234 /usr/local/bin/loki -config.file=/etc/loki/loki.yaml ...Verify the /ready endpoint returns 200:
curl http://localhost:3100/readyExpected output:
readyAnd /metrics should return Prometheus-format metrics:
curl -s http://localhost:3100/metrics | head -20Step 7: Install and Configure Promtail
Loki does not collect logs itself — it is a server that receives logs pushed to it. Promtail is the official agent that tails log files on each host, attaches labels, and forwards them to Loki over HTTP.
Install Promtail on the same server as Loki first. Later you will deploy Promtail to every host whose logs you want to collect.
cd /tmp wget "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/promtail-linux-amd64.zip" unzip promtail-linux-amd64.zip sudo mv promtail-linux-amd64 /usr/local/bin/promtail sudo chmod +x /usr/local/bin/promtail
sudo groupadd --system promtail sudo useradd --system --no-create-home --shell /usr/sbin/nologin --gid promtail promtail sudo usermod -aG adm promtail # access to /var/log/syslog and similar sudo mkdir -p /etc/promtail /var/lib/promtail sudo chown -R promtail:promtail /var/lib/promtail
Create /etc/promtail/promtail.yaml:
sudo tee /etc/promtail/promtail.yaml > /dev/null <<'EOF' server: http_listen_port: 9080 grpc_listen_port: 0positions: filename: /var/lib/promtail/positions.yaml
clients: - url: http://localhost:3100/loki/api/v1/push
scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: varlogs host: web-01 __path__: /var/log/*.log
- job_name: journal journal: max_age: 12h labels: job: systemd-journal host: web-01 path: /var/log/journal relabel_configs: - source_labels: ['__journal__systemd_unit'] target_label: 'unit' - source_labels: ['__journal_priority_keyword'] target_label: 'level'
- job_name: nginx static_configs: - targets: - localhost labels: job: nginx host: web-01 __path__: /var/log/nginx/*.log pipeline_stages: - match: selector: '{job="nginx"}' stages: - regex: expression: '^(?P<remote_addr>[\d\.]+) - (?P<remote_user>\S+) \[(?P<time_local>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<protocol>\S+)" (?P<status>\d+) (?P<body_bytes_sent>\d+)' - labels: method: status: EOF
sudo chown promtail:promtail /etc/promtail/promtail.yaml
The config has three scrape jobs:
.log file in /var/log/ and labels it job=varlogs.Create the Promtail systemd unit:
sudo tee /etc/systemd/system/promtail.service > /dev/null <<'EOF' [Unit] Description=Promtail log shipper Documentation=https://grafana.com/docs/loki/latest/send-data/promtail/ Wants=network-online.target After=network-online.target[Service] Type=simple User=promtail Group=promtail ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail.yaml Restart=on-failure RestartSec=10
[Install] WantedBy=multi-user.target EOF
sudo systemctl daemon-reload sudo systemctl enable --now promtail sudo systemctl status promtail
Within 30 seconds, Promtail will start pushing logs to Loki. Verify:
logcli --addr=http://localhost:3100 labelsExpected output:
host
job
level
method
status
unitStep 8: Connect Loki as a Grafana Datasource
Assuming you already have Grafana running (on the same host or a different one), add Loki as a datasource.
In the Grafana UI, navigate to Connections -> Data sources -> Add data source, select Loki, and configure:
- Name:
Loki - URL:
http://localhost:3100(or the internal IP of your Loki server) - HTTP Header (optional):
X-Scope-OrgID: fakeifauth_enabled: true
Alternatively, provision the datasource via YAML (the 12-factor approach):
# /etc/grafana/provisioning/datasources/loki.yaml apiVersion: 1
datasources: - name: Loki type: loki access: proxy url: http://loki.internal:3100 jsonData: maxLines: 5000 timeout: 60
Restart Grafana, and the datasource appears automatically. Open the Explore view, pick Loki, and run your first query — {job="varlogs"} should return log lines from /var/log.
Step 9: Query Logs with LogQL
LogQL is the query language for Loki, designed to feel familiar if you know PromQL. Every query has two parts: a log stream selector (label matchers) and an optional filter expression.
Basic Stream Selection
{job="varlogs"}Returns all log lines with the label job=varlogs.
{job="nginx", status="500"}Returns all Nginx log lines where the parsed status was 500.
Filtering Log Content
After the selector, pipe the stream through line filters:
{job="nginx"} |= "GET /api"|= means "line contains". Other operators:
!=— line does not contain|~— line matches regex!~— line does not match regex
/api/users:{job="nginx", status=~"5.."} |= "/api/users"Parser Stages
Chain parsers to extract fields at query time:
{job="app"}
| json
| duration_ms > 1000
| line_format "{{.request_id}} took {{.duration_ms}}ms"Aggregation (Metric Queries)
LogQL can convert log streams into time-series metrics. This is Loki's killer feature — you can alert on log patterns without running a separate SIEM.
Count 500 errors per minute by host:
sum by (host) (rate({job="nginx", status="500"}[1m]))95th percentile request latency if you emit a duration_ms field:
quantile_over_time(0.95,
{job="nginx"}
| json
| unwrap duration_ms [5m]
) by (path)Common Useful Queries
# All error-level logs in the last hour
{job="systemd-journal", level="err"}SSH failed-login attempts
{unit="ssh.service"} |= "Failed password"Top 10 noisiest systemd units
topk(10, sum by (unit) (count_over_time({job="systemd-journal"}[1h])))Any log with stack traces
{job=~".+"} |~ "Traceback|Exception|panic:"Run any of these from the command line with logcli:
logcli --addr=http://localhost:3100 query '{job="nginx", status="500"}' --limit 50Step 10: Configure Retention Policies
By default, the limits_config.retention_period in loki.yaml applies globally. To set per-tenant or per-stream retention, use the retention_stream section of limits_config.
Edit /etc/loki/loki.yaml:
limits_config: retention_period: 744h # 31 days default retention_stream: - selector: '{job="nginx"}' priority: 1 period: 2160h # 90 days for Nginx - selector: '{level="debug"}' priority: 2 period: 72h # 3 days for debug logs - selector: '{job="audit"}' priority: 1 period: 8760h # 1 year for audit logs
compactor: working_directory: /var/lib/loki/compactor compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 delete_request_store: s3
Lower priority wins when multiple stream selectors match. After changes, restart Loki:
sudo systemctl restart lokiThe compactor runs every 10 minutes and marks chunks older than the retention window for deletion. Actual deletion from S3 happens after retention_delete_delay, giving you a 2-hour safety window to recover from misconfiguration.
Monitor the compactor:
curl -s http://localhost:3100/metrics | grep loki_compactor_Key metrics to watch:
loki_compactor_oldest_pending_delete_request_age_seconds— should be low; high values mean the compactor is falling behind.loki_compactor_deleted_chunks_total— counter of chunks successfully deleted.
Performance Tuning
A monolithic Loki on a Professional VPS (6 vCPU / 12 GB) comfortably handles 50-100 GB/day. Push beyond that and you need to tune.
Ingester Flush Behavior
The ingester buffers log chunks in memory and flushes them to S3 when any of these conditions trigger:
- Chunk reaches
chunk_target_size(default 1.5 MB) - Chunk is idle for
chunk_idle_period - Chunk is older than
max_chunk_age
chunk_target_size to 3 MB. For ultra-low-latency tail -f in Grafana, decrease chunk_idle_period to 5 minutes.Query Parallelism
The query-frontend splits long queries into sub-queries and runs them in parallel:
limits_config:
max_query_parallelism: 32
split_queries_by_interval: 30mFor a query spanning 24 hours with split_queries_by_interval: 30m, Loki runs 48 sub-queries capped at 32 in parallel. Higher parallelism speeds up long-range queries but increases S3 request costs.
Caching
Enable results caching for repeated dashboard queries:
query_range:
align_queries_with_step: true
cache_results: true
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 512
ttl: 24hOn high-traffic deployments, swap the embedded cache for external Memcached or Redis.
Ingestion Rate Limits
Protect Loki from runaway log sources:
limits_config:
ingestion_rate_mb: 16 # per tenant MB/s
ingestion_burst_size_mb: 32 # short-burst allowance
per_stream_rate_limit: 5MB
per_stream_rate_limit_burst: 15MBIf Promtail starts reporting 429 Too Many Requests, raise these limits — but also investigate whether a misbehaving application is the real cause.
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
context deadline exceeded on S3 writes | Network latency or wrong endpoint | Verify the S3 endpoint URL. Test with aws s3 ls s3://loki-chunks --endpoint-url=.... Increase storage_config.aws.http_config.idle_conn_timeout. |
no such host for S3 endpoint | DNS resolution or typo | Resolve the hostname manually: dig s3.eu-central-1.amazonaws.com. Check /etc/resolv.conf. |
Promtail shows 429 Too Many Requests | Loki ingestion rate limit hit | Increase limits_config.ingestion_rate_mb and per_stream_rate_limit. Check for runaway log producers. |
| Queries return empty results but labels exist | Time range mismatch | In Grafana, set the time range to include a period when logs were pushed. Check positions.yaml in Promtail — it may be tailing from the end of the file. |
failed to flush user in Loki logs | S3 credentials invalid or bucket unreachable | Check credentials in /etc/loki/loki.env. Verify bucket policy allows the IAM user to PUT objects. |
Disk fills up on /var/lib/loki | WAL or tsdb-shipper cache growing | Check df -h. If WAL is large, it means S3 writes are failing. Fix S3 connectivity; then WAL drains automatically. |
| Retention not deleting old logs | Compactor disabled or not running | Verify compactor.retention_enabled: true. Check loki_compactor_runs_completed_total is incrementing. |
msg="failed to create index client" | Wrong schema version | Schema v13 requires Loki 3.x. Older Loki uses v11 or v12. Never delete an existing schema period. |
| High memory usage | Too many active streams (label cardinality) | Check loki_ingester_memory_streams. Reduce label cardinality — avoid labels with high-cardinality values like user IDs or request paths. |
level=error msg="error writing WAL" | Disk full or permissions | Check df -h and ownership of /var/lib/loki/wal. Must be writable by the loki user. |
Viewing Logs
Loki's own logs are your best debugging tool:
sudo journalctl -u loki -f
sudo journalctl -u promtail -fFAQ
How does Loki compare to Elasticsearch for log search?
Loki indexes only labels (host, job, namespace, level) and stores log content as compressed chunks in object storage. Elasticsearch indexes every word in every log line as part of an inverted index, enabling fast full-text search, fuzzy matching, and aggregations over arbitrary fields. Loki is 3-10x cheaper to run for the same ingestion volume and simpler to operate, but Elasticsearch wins for ad-hoc exploratory search where you do not know which labels to query in advance. For most observability use cases — "show me errors from service X between 10:00 and 10:15" — LogQL with label selectors and grep-style filters is plenty fast and dramatically cheaper.
Can I migrate from Graylog or ELK to Loki without data loss?
Historical logs in Graylog or Elasticsearch cannot be directly imported into Loki — the storage formats are incompatible. The practical migration path is to run both systems in parallel for the retention period of your old system (typically 30-90 days), forward new logs to both Loki and the legacy system during that window, and decommission the legacy system once its retention window has rolled over. Graylog users often appreciate Loki's simpler operational model — no Elasticsearch, no MongoDB, no JVM tuning — once the migration completes.
Does Loki require S3, or can I use local disk only?
Loki supports local filesystem storage via the filesystem backend, which is useful for development and testing. For production, always use object storage (S3, GCS, Azure Blob, or self-hosted MinIO). Local disk storage does not scale horizontally, does not survive instance replacement, and limits retention to whatever fits on the local volume. Even on a single-node deployment, pointing at an S3-compatible object store gives you durability, cheap long-term retention, and a trivial upgrade path to Simple Scalable mode.
How do I secure the Loki HTTP endpoint?
Loki has no built-in authentication. In production, put it behind a reverse proxy (Nginx, Traefik, or Caddy) with TLS and basic auth or OAuth2 Proxy. For multi-tenant deployments, set auth_enabled: true in loki.yaml and have the auth proxy inject the X-Scope-OrgID header based on the authenticated user. Also firewall ports 3100 and 9096 so only Promtail agents and Grafana can reach them — never expose Loki directly to the public internet.
What labels should I use in Promtail?
Keep label cardinality low. Good labels: host, job, env, namespace, cluster, unit. Bad labels: request_id, user_id, trace_id, path, ip — any value that changes often creates a new stream, and Loki's ingester keeps one in-memory buffer per stream. A deployment with 10,000+ active streams per tenant typically indicates a cardinality problem that will eventually OOM the ingester. Put high-cardinality values inside the log line itself (as JSON or key=value pairs) and extract them at query time with LogQL's | json and | logfmt parsers, which do not create new streams.
Can Loki alert on log patterns like Elasticsearch Watcher?
Yes. Loki's ruler component supports alerting rules written in LogQL, compatible with Prometheus Alertmanager. For example, you can alert when the rate of 500 errors exceeds 5 per minute, or when a specific error message appears more than 10 times in 5 minutes. Rules live in YAML files in the directory pointed to by ruler.storage.local.directory and Loki evaluates them on a configurable interval. Alerts are pushed to the same Alertmanager you use for metric alerts, giving you unified alert routing and deduplication across logs and metrics.
How much does a self-hosted Loki deployment actually cost?
For a team ingesting around 50 GB/day, you need a CloudCore Professional VPS (EUR 19.99/month) plus object storage costs. On AWS S3 with 30-day retention and lz4 compression, that is roughly 450 GB stored at $0.023/GB = ~$10/month plus PUT requests (~$3/month). Backblaze B2 or Wasabi drop that to under $3/month. Total monthly cost: EUR 25-30. An equivalent Elasticsearch-based stack needs at least a Business-tier VPS (EUR 39.99/month) with no way to offload cold data to cheap storage without additional complexity.
Next Steps
Now that Loki is running on your VPS, here are recommended follow-ups to build out a complete observability stack:
- Install Grafana and build log dashboards — If you have not already, install Grafana on the same VPS and build dashboards that combine Loki log panels with Prometheus metric panels for side-by-side troubleshooting.
- Deploy Promtail to every host — Install Promtail on every server, container host, and Kubernetes cluster whose logs you want to aggregate. Use the same label scheme across all Promtails so queries can span hosts.
- Add Prometheus for metrics — Install Prometheus alongside Loki and scrape the
/metricsendpoints of both Loki and Promtail. This gives you visibility into ingestion rates, query latency, and compaction progress, and lets you alert on Loki itself falling behind.
- Set up MinIO for self-hosted object storage — If AWS S3 egress costs concern you, deploy MinIO on a separate VPS with bulk storage and point Loki at it. You get S3-compatible storage with zero egress fees.
- Migrate from Elasticsearch or Graylog — If you are running Elasticsearch or Graylog, start dual-writing logs to Loki today and retire the heavier stack once your legacy retention window rolls over.
- Read the official Loki docs — The Grafana Labs documentation at grafana.com/docs/loki/latest/ is the authoritative reference for configuration options, LogQL functions, and scaling guidance.
Need more storage or CPU? Upgrade your VPS at any time from the Professional plan to a higher tier without re-provisioning. Loki's chunk storage lives in S3, so scaling up takes only a resize and a reboot.