How to Install ELK Stack (Elasticsearch, Logstash, Kibana) on Ubuntu 24.04

Centralised log management used to mean either paying Datadog Logs six figures a year, feeding Splunk a budget the size of a small SaaS, or assembling a fragile mess of grep, rsyslog and shell scripts. The ELK Stack — now officially called the Elastic Stack — changed that by giving you an open, self-hostable platform for logs, metrics, traces, and security events. This guide walks through a complete production install of Elastic Stack 8.x on a single Ubuntu 24.04 VPS: Elasticsearch as the search and storage engine, Kibana as the UI, Logstash as the pipeline, and Filebeat as the shipper. By the end you will have TLS-secured ingestion, working dashboards, index lifecycle management, snapshots, and an Nginx reverse proxy — everything you need to ingest gigabytes of logs a day without handing a credit card to a SaaS vendor.

Skip the setup? Deploy the ELK Stack in one click with our pre-configured Elastic Stack image. Launch an observability-ready VPS now and start ingesting logs in under five minutes.

Table of Contents

What is the ELK / Elastic Stack?

The name "ELK" is a three-letter mnemonic for the original core of the stack: Elasticsearch, Logstash, and Kibana. Elastic added Beats (a family of lightweight log/metric shippers) in 2015, and since then the company has rebranded the product family as the Elastic Stack to reflect the broader scope. In practice, engineers still use "ELK" interchangeably, and most greenfield deployments today include all four components.

Each piece plays a distinct role:

• Elasticsearch is a distributed JSON document store built on top of Apache Lucene. It indexes every field of every document, supports full-text search, aggregations, vector search, and scales horizontally by sharding indices across nodes.
• Logstash is a JVM-based event pipeline. It ingests data from dozens of sources (files, Kafka, Syslog, JDBC, HTTP), transforms it with filter plugins (grok, mutate, geoip, user-agent, date parsing), and ships it to Elasticsearch or elsewhere.
• Kibana is the web UI. It provides dashboards, Discover (a SQL-for-logs explorer), Lens (drag-and-drop visualisations), alerting rules, the SIEM app, machine learning jobs, and administrative tooling for managing indices, users, and spaces.
• Beats are single-purpose Go binaries that ship data from edge hosts. Filebeat tails log files, Metricbeat collects system and service metrics, Packetbeat sniffs network traffic, Auditbeat watches audit events, Heartbeat performs uptime checks, and Winlogbeat handles Windows Event Log.

Why Self-Host ELK?

Running your own Elastic Stack versus paying for Datadog Logs, Splunk, or Elastic Cloud is primarily an economics and sovereignty decision.

• Cost at volume -- Datadog Logs bills roughly $1.70 per GB ingested and $1.40 per million events retained. A team pushing 50 GB of logs per day pays $2,500+/month. Splunk Enterprise starts around $2,000/GB/day for ingest licences. A self-hosted ELK box ingesting the same 50 GB/day runs comfortably on a €35.99/month VPS.
• No retention taxes -- Hosted log SaaS charges you separately for storage beyond 7 or 15 days. Self-hosted ELK lets you keep 90 days of hot data and archive indefinitely to S3-compatible storage at whatever rate your object store charges.
• Data residency -- For GDPR, HIPAA, or sovereignty-sensitive workloads, keeping logs on an EU VPS you control is simpler than negotiating DPAs with a US SaaS. You decide exactly where the disks live.
• Full query power -- Elasticsearch's Query DSL, aggregations, runtime fields, and ES|QL give you more depth than the restricted search languages most SaaS vendors expose. You can run complex joins, anomaly detection, and ML jobs on any field.
• No sampling -- Hosted APM tools sample traces and drop "noisy" logs to stay under your billing tier. Your own ELK ingests 100% of events; you keep all the evidence when an incident hits.
• Integrated SIEM -- The Kibana SIEM app and pre-built detection rules rival commercial tools like LogRhythm or QRadar. Teams regularly replace €50k/year SIEM appliances with a self-hosted ELK cluster.
• Custom enrichment -- You can add any enrichment pipeline (GeoIP, WHOIS, threat intel feeds, internal CMDB lookups) without paying per-integration fees.

Cost Comparison: Self-Hosted vs Hosted Log Platforms

Licensing: SSPL, Elastic License, and OpenSearch

It is worth understanding Elastic's licensing before committing, because it shaped the observability ecosystem.

In January 2021 Elastic relicensed Elasticsearch and Kibana from Apache 2.0 to a dual licence: the Server Side Public License (SSPL) and the Elastic License v2 (ELv2). The change specifically targets cloud providers that resell Elasticsearch as a managed service without contributing upstream. Self-hosting the stack on your own servers — exactly what this guide does — is allowed for free, including for commercial products. You cannot, however, turn around and sell Elasticsearch-as-a-Service to third parties.

In response, AWS forked Elasticsearch 7.10 and created OpenSearch, which stays on Apache 2.0 and is maintained by the Linux Foundation. OpenSearch and OpenSearch Dashboards are API-compatible with Elasticsearch 7.10 but have diverged since. If your employer's legal team refuses SSPL outright, OpenSearch is a drop-in alternative — most of this guide's commands work with small adjustments (different repo URL, opensearch package names, no Fleet/Agent, different ML features). For the majority of users, Elastic Stack 8.x's free tier is more feature-complete (vector search, ES|QL, APM, SIEM detection rules) and we recommend it for most self-hosted deployments.

The Elastic Stack also has paid subscription tiers (Gold, Platinum, Enterprise) that unlock SSO, field-level security, cross-cluster replication, and commercial support. Everything covered in this guide runs on the free Basic tier bundled with the default install.

Architecture Overview

A typical ELK deployment looks like this:

Edge hosts                  Logstash                Elasticsearch            Kibana
  (web/app/DB servers)        (pipelines/             (storage,                (UI, dashboards,
                               enrichment)             search, ML)              alerts, SIEM)
  +---------+                 +----------+            +------------+           +---------+
  | nginx   | -- Filebeat --> | :5044    | --> :9200->| shards,    |<-- :9200->| :5601   |<-- :443 Nginx
  | app     |    (beats       | grok,    |  TLS+auth  | aggregations|           | Lens,   |
  | docker  |    protocol)    | geoip,   |            | ILM, ML    |           | SIEM,   |
  +---------+                 | mutate   |            +------------+           | rules   |
                              +----------+                                      +---------+
                                   |
                                   +-- dead letter queue / Kafka (optional)

For small-to-medium workloads, all four components can co-exist on a single VPS. As you scale, the usual progression is:

You can also skip Logstash entirely and use Elasticsearch's native ingest pipelines for lightweight parsing. Logstash shines when you need heavy enrichment, multi-output fan-out, persistent queues, or non-Elastic outputs (Kafka, S3, PostgreSQL).

Prerequisites

Before you begin, make sure you have:

• A VPS running Ubuntu 24.04 LTS with root or sudo access
• At least 4 vCPU and 8 GB of RAM — Elasticsearch is RAM-hungry. 8 GB is the practical minimum for a single-node dev/small-prod deployment; 16 GB is recommended.
• At least 80 GB of SSD storage — logs compound quickly. Plan for 7-14 days of hot storage on fast disks.
• A domain name (e.g. kibana.yourdomain.com) pointing to your VPS IP, for the Nginx reverse proxy and TLS certificates.
• Open inbound ports from your edge hosts: TCP/5044 (Logstash Beats input), TCP/9200 (Elasticsearch, only if direct ingest), TCP/443 (Kibana via Nginx).

Recommended Plan: CloudCore Business

Elasticsearch eats RAM for breakfast. For a comfortable single-node ELK box handling a few dozen GB/day, we recommend the CloudCore Business plan:

- 6 vCPU cores

- 16 GB RAM

- 200 GB NVMe SSD

- Unmetered bandwidth

- EUR 35.99/month

That gives you a 7-8 GB JVM heap for Elasticsearch with headroom for Kibana (~1.5 GB), Logstash (~1 GB), and the kernel page cache (critical for Lucene segment performance). For larger multi-node clusters, scale up or deploy separate boxes per role.

Connect to your server via SSH to begin:

ssh root@your-server-ip

Step 1: Prepare the Server

Update packages, install dependencies, and apply a few Elasticsearch-specific kernel tweaks.

sudo apt update && sudo apt upgrade -y
sudo apt install -y apt-transport-https ca-certificates curl gnupg openjdk-17-jre-headless ufw

Elasticsearch 8.x bundles its own OpenJDK — installing the system JDK is optional but useful for Logstash and general JVM tooling.

Disable Swap

Swapping is catastrophic for Elasticsearch performance because Lucene relies on keeping hot index segments in the filesystem cache. Disable swap:

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Verify:

free -h

Expected output should show Swap: 0B 0B 0B.

If you cannot disable swap entirely (some providers require it), set bootstrap.memory_lock: true in elasticsearch.yml later and configure systemd LimitMEMLOCK=infinity.

Raise Virtual Memory Limits

Elasticsearch memory-maps Lucene index files, which requires a higher vm.max_map_count than the default:

sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

Basic Firewall

sudo ufw allow OpenSSH
sudo ufw allow 443/tcp
sudo ufw allow 80/tcp
sudo ufw allow from YOUR_EDGE_HOST_IP to any port 5044 proto tcp
sudo ufw enable

Replace YOUR_EDGE_HOST_IP with the IP range of the servers that will ship logs via Filebeat. Do not expose 9200 or 5044 to the public internet.

Step 2: Add the Elastic APT Repository

Import Elastic's GPG key and register the 8.x APT repository.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

Expected output:

Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Get:2 https://artifacts.elastic.co/packages/8.x/apt stable InRelease [10.4 kB]
Get:3 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 Packages [42.7 kB]
...
Reading package lists... Done

All four components (Elasticsearch, Kibana, Logstash, Filebeat) come from this single repository.

Step 3: Install and Configure Elasticsearch

Install the Elasticsearch package:

sudo apt install -y elasticsearch

During the install you will see a critical block of output — capture it:

--------------------------- Security autoconfiguration information ------------------------------
Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.
The generated password for the elastic built-in superuser is : rT3k9Xc_W2mPqLa7FpZ8
If this node should join an existing cluster, you can reconfigure this with
  /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>
after creating an enrollment token on your existing cluster.
You can complete the following actions at any time:
  Reset the password of the elastic built-in superuser with
  '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.
  Generate an enrollment token for Kibana instances with
   '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.
-------------------------------------------------------------------------------------------------

The elastic superuser password and the enrollment tooling are the key artefacts. Store the password in a password manager now — re-displaying it requires a reset.

Edit the main configuration file:

sudo nano /etc/elasticsearch/elasticsearch.yml

Set the following values (leaving the auto-generated xpack.security.* section intact at the bottom):

# ======================== Elasticsearch Configuration =========================
cluster.name: elk-prod
node.name: elk-node-01
---------------------------------- Paths -------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
---------------------------------- Network -----------------------------------
network.host: 127.0.0.1
http.port: 9200
--------------------------------- Discovery ----------------------------------
discovery.type: single-node
---------------------------------- Various -----------------------------------
bootstrap.memory_lock: false
action.destructive_requires_name: true

Key settings explained:

• cluster.name -- An identifier; nodes with the same cluster name will try to join each other, so pick something distinctive.
• node.name -- Human-readable node identifier, shown in Kibana and logs.
• network.host: 127.0.0.1 -- Bind only to localhost. Kibana, Logstash, and Filebeat will reach ES through the loopback interface; external access is never required for this topology. If you later move Kibana to a separate server, change this to the private network IP and lock port 9200 with UFW.
• discovery.type: single-node -- Tells Elasticsearch not to expect peers. Without this, a single-node install waits forever for a quorum. For multi-node clusters, remove this and configure discovery.seed_hosts + cluster.initial_master_nodes.
• action.destructive_requires_name: true -- Prevents wildcard deletes like DELETE /* from nuking all your indices. Absolutely worth it.

Step 4: Tune the JVM Heap

By default Elasticsearch autosizes heap based on node RAM, but on shared VPS hardware you want explicit control. The rule of thumb is: set heap to 50% of physical RAM, never above 30-32 GB (above that, pointer compression breaks and you lose memory efficiency).

Create a heap override file:

sudo mkdir -p /etc/elasticsearch/jvm.options.d
sudo tee /etc/elasticsearch/jvm.options.d/heap.options > /dev/null <<EOF
-Xms8g
-Xmx8g
EOF

Sizing guidance:

Step 5: Start Elasticsearch and Test

Enable the service to start on boot, then start it:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Elasticsearch takes 20-60 seconds to come up. Watch the logs:

sudo journalctl -u elasticsearch -f

Look for started and recovered [0] indices into cluster_state. Press Ctrl+C when you see them.

Check status:

sudo systemctl status elasticsearch

Test the HTTPS endpoint with the elastic password:

curl -k -u elastic:YOUR_ELASTIC_PASSWORD https://localhost:9200

Expected output:

{
  "name" : "elk-node-01",
  "cluster_name" : "elk-prod",
  "cluster_uuid" : "7h_aBcD12QEFgh...",
  "version" : {
    "number" : "8.15.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "lucene_version" : "9.11.1"
  },
  "tagline" : "You Know, for Search"
}

The -k flag accepts the self-signed TLS certificate the installer generated. We will address proper certificates in Step 10.

If you lost the elastic password, reset it:

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Step 6: Install and Configure Kibana

Install the Kibana package:

sudo apt install -y kibana

Generate an enrollment token — this is how Kibana authenticates to Elasticsearch the first time:

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Copy the long base64 token it prints. You will paste it into the Kibana web UI on first run.

Also reset the kibana_system user password (the internal service account):

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system

Record this password too.

Edit /etc/kibana/kibana.yml:

sudo nano /etc/kibana/kibana.yml

Set:

server.port: 5601
server.host: "127.0.0.1"
server.publicBaseUrl: "https://kibana.yourdomain.com"
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "YOUR_KIBANA_SYSTEM_PASSWORD"
The ES self-signed CA is placed here by the installer when Kibana is enrolled.
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"]
Encryption keys — generate with: openssl rand -hex 32
xpack.encryptedSavedObjects.encryptionKey: "CHANGE_ME_32_BYTE_HEX"
xpack.reporting.encryptionKey: "CHANGE_ME_32_BYTE_HEX"
xpack.security.encryptionKey: "CHANGE_ME_32_BYTE_HEX"
logging.root.level: info

Generate three random 32-byte hex keys and paste them into the three encryptionKey fields — if you skip this, every Kibana restart invalidates session cookies, saved objects, and scheduled reports:

for i in 1 2 3; do openssl rand -hex 32; done

Start Kibana:

sudo systemctl daemon-reload
sudo systemctl enable kibana
sudo systemctl start kibana

Kibana's first boot runs migrations and takes 1-2 minutes. Tail the logs:

sudo journalctl -u kibana -f

When you see Kibana is now available (was degraded), it is ready.

Step 7: Reverse Proxy Kibana with Nginx

Kibana's built-in HTTP server is fine for localhost, but you want TLS terminated by Nginx with a real certificate and HTTP/2.

Install Nginx and Certbot:

sudo apt install -y nginx certbot python3-certbot-nginx

Create the site configuration:

sudo tee /etc/nginx/sites-available/kibana > /dev/null <<'EOF'
server {
    listen 80;
    server_name kibana.yourdomain.com;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl http2;
    server_name kibana.yourdomain.com;
    ssl_certificate     /etc/letsencrypt/live/kibana.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/kibana.yourdomain.com/privkey.pem;
    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options SAMEORIGIN;
    client_max_body_size 100m;
    location / {
        proxy_pass http://127.0.0.1:5601;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/
sudo certbot --nginx -d kibana.yourdomain.com
sudo nginx -t && sudo systemctl reload nginx

Visit https://kibana.yourdomain.com. You will be prompted to paste the enrollment token from Step 6, then log in with the elastic superuser and its password. Welcome to Kibana.

Step 8: Install and Configure Logstash

Logstash is optional if you only need simple ingest, but most real pipelines benefit from its filter ecosystem.

sudo apt install -y logstash

Create a pipeline for Beats input → grok parsing → Elasticsearch output:

sudo tee /etc/logstash/conf.d/10-beats-to-es.conf > /dev/null <<'EOF'
input {
  beats {
    port => 5044
    ssl_enabled => true
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.key"
  }
}
filter {
  if [event][module] == "nginx" and [event][dataset] == "nginx.access" {
    grok {
      match => { "message" => "%{IPORHOST:[client][ip]} - %{DATA:[user][name]} \[%{HTTPDATE:[@metadata][timestamp]}\] \"%{WORD:[http][request][method]} %{DATA:[url][original]} HTTP/%{NUMBER:[http][version]}\" %{NUMBER:[http][response][status_code]:int} %{NUMBER:[http][response][body][bytes]:int} \"%{DATA:[http][request][referrer]}\" \"%{DATA:[user_agent][original]}\"" }
    }
    date {
      match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "[client][ip]"
      target => "[client][geo]"
    }
    useragent {
      source => "[user_agent][original]"
      target => "[user_agent]"
    }
  }
  mutate {
    remove_field => ["message", "host"]
  }
}
output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    ssl_enabled => true
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    api_key => "YOUR_API_KEY_ID:YOUR_API_KEY_SECRET"
    data_stream => true
    data_stream_type => "logs"
    data_stream_dataset => "%{[event][dataset]}"
    data_stream_namespace => "prod"
  }
}
EOF

Generate an API Key for Logstash

Logging in as elastic for pipeline output is bad hygiene. Create a least-privilege API key in Kibana: Stack Management → API keys → Create API key, then grant monitor_cluster and write/create_index on logs--.

Alternatively, via curl:

curl -k -u elastic:YOUR_PASSWORD -X POST "https://localhost:9200/_security/api_key" \
  -H 'Content-Type: application/json' -d '{
    "name": "logstash-writer",
    "role_descriptors": {
      "logstash_writer": {
        "cluster": ["monitor"],
        "index": [{
          "names": ["logs--"],
          "privileges": ["create_index", "create", "write", "auto_configure"]
        }]
      }
    }
  }'

The response gives you an id and api_key — paste id:api_key into the api_key field in the pipeline config.

Enable and start Logstash:

sudo systemctl enable logstash
sudo systemctl start logstash
sudo journalctl -u logstash -f

Look for Beats inputs: Starting input listener and Successfully started Logstash API endpoint.

JVM Tuning for Logstash

Logstash also runs on the JVM. For a 16 GB server, give it 1-2 GB:

sudo sed -i 's/^-Xms.*/-Xms1g/' /etc/logstash/jvm.options
sudo sed -i 's/^-Xmx.*/-Xmx1g/' /etc/logstash/jvm.options
sudo systemctl restart logstash

Step 9: Install Filebeat on a Remote Host

On each edge server (web servers, app servers, DB servers), install Filebeat to ship logs.

# Run on the edge host, NOT the ELK server
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install -y filebeat

Enable the modules you need:

sudo filebeat modules enable system nginx auth
sudo filebeat modules list

Configure /etc/filebeat/filebeat.yml:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s
Ship via Logstash (recommended when you need enrichment)
output.logstash:
  hosts: ["elk.yourdomain.com:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
OR ship directly to Elasticsearch (skip Logstash for simple ingest)
output.elasticsearch:
  hosts: ["https://elk.yourdomain.com:9200"]
  api_key: "YOUR_API_KEY_ID:YOUR_API_KEY_SECRET"
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  keepfiles: 7

Configure the nginx module at /etc/filebeat/modules.d/nginx.yml:

- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]

If you ship directly to Elasticsearch, load the index templates and Kibana dashboards once from the edge host:

sudo filebeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['elk.yourdomain.com:9200'] \
  -E output.elasticsearch.api_key="ID:KEY" \
  -E setup.kibana.host=kibana.yourdomain.com:443 \
  -E setup.kibana.protocol=https

Start Filebeat:

sudo systemctl enable filebeat
sudo systemctl start filebeat
sudo journalctl -u filebeat -f

Within 60 seconds you will see events flowing into Kibana → Discover → logs-nginx.access-*.

Step 10: TLS Across the Stack

Elasticsearch 8.x auto-generated a self-signed CA and node certificates at install time. For production you should issue proper certificates with known expiry.

Generate a cluster CA:

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca \
  --out /etc/elasticsearch/certs/elastic-stack-ca.p12 \
  --pass ''

Generate a node certificate signed by that CA:

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert \
  --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 \
  --ca-pass '' \
  --out /etc/elasticsearch/certs/elastic-certificates.p12 \
  --pass '' \
  --dns elk.yourdomain.com,localhost \
  --ip 127.0.0.1

Reference the certificates in elasticsearch.yml:

xpack.security.enabled: true
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/elastic-certificates.p12
  truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/elastic-certificates.p12

Fix permissions and restart:

sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch/certs
sudo chmod 640 /etc/elasticsearch/certs/*.p12
sudo systemctl restart elasticsearch

Extract the CA cert as PEM for clients (Logstash, Filebeat, Kibana) that need it:

sudo openssl pkcs12 -in /etc/elasticsearch/certs/elastic-stack-ca.p12 \
  -clcerts -nokeys -out /etc/elasticsearch/certs/ca.pem

Copy ca.pem to each client's ssl.certificate_authorities path.

Step 11: Index Lifecycle Management (ILM)

ILM automatically rolls indices through hot → warm → cold → delete phases based on age or size. Without ILM, Elasticsearch indices grow unbounded and eventually trigger a disk watermark flood.

Create a policy in Kibana → Stack Management → Index Lifecycle Policies → Create policy, or via the API:

curl -k -u elastic:PASS -X PUT "https://localhost:9200/_ilm/policy/logs-90d" \
  -H 'Content-Type: application/json' -d '{
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": { "max_primary_shard_size": "50gb", "max_age": "7d" },
            "set_priority": { "priority": 100 }
          }
        },
        "warm": {
          "min_age": "7d",
          "actions": {
            "shrink": { "number_of_shards": 1 },
            "forcemerge": { "max_num_segments": 1 },
            "set_priority": { "priority": 50 }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {
            "set_priority": { "priority": 0 },
            "searchable_snapshot": { "snapshot_repository": "backups" }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": { "delete": {} }
        }
      }
    }
  }'

Attach it via an index template:

curl -k -u elastic:PASS -X PUT "https://localhost:9200/_index_template/logs-template" \
  -H 'Content-Type: application/json' -d '{
    "index_patterns": ["logs--"],
    "data_stream": {},
    "template": {
      "settings": {
        "index.lifecycle.name": "logs-90d",
        "index.number_of_shards": 1,
        "index.number_of_replicas": 0
      }
    }
  }'

On a single-node cluster, always set number_of_replicas: 0 — replicas require a second node, and Elasticsearch will mark the cluster YELLOW forever without one.

Step 12: Dashboards, Alerting, and SIEM

Import Pre-Built Dashboards

Filebeat modules ship with Kibana dashboards. After the first filebeat setup run, navigate to Kibana → Analytics → Dashboards and filter by tag nginx, system, or auth. You will get working dashboards for request rates, error ratios, top URLs, GeoIP maps, and failed login attempts out of the box.

Alerting Rules

Kibana → Stack Management → Rules and Connectors. Create rules like:

• Elasticsearch query rule: Alert when http.response.status_code >= 500 count exceeds 20 in 5 minutes.
• Index threshold rule: Alert when average system.cpu.total.pct > 0.9 for any host over 10 minutes.
• Anomaly detection rule: Connect to a Machine Learning job (Basic tier includes ML alerting in 8.x) and alert on anomaly scores above 75.

SIEM App

Kibana includes a full Security app at Security → Overview. It ingests logs-* indices that follow the Elastic Common Schema (ECS) and provides:

• Hosts and network event timelines
• 700+ pre-built detection rules (Mitre ATT&CK mapped)
• Case management for investigations
• Osquery integration for live endpoint queries
• Timeline view for incident reconstruction

Step 13: RBAC — Roles, Users, and Spaces

Never share the elastic superuser account. Create dedicated roles and users.

Create a Read-Only Role via Kibana

Stack Management → Security → Roles → Create role:

• Name: logs-viewer
• Cluster privileges: monitor
• Index privileges: indices logs-*, privileges read, view_index_metadata
• Kibana privileges: space Default, feature Analytics Read, feature Discover Read, feature Dashboards Read

Create a User

Stack Management → Security → Users → Create user: assign the logs-viewer role.

Spaces

Spaces are multi-tenant sub-UIs within the same Kibana install. Use them to isolate dashboards and index access per team — e.g. a platform space for SRE and a security space for the SOC. Role mappings limit what each user sees per space.

Step 14: Snapshots and Backups

Elasticsearch's native backup mechanism is snapshots — incremental, file-level backups written to a repository.

Register a Filesystem Repository

Add the backup path to elasticsearch.yml:

path.repo: ["/var/backups/elasticsearch"]

Create the directory and restart:

sudo mkdir -p /var/backups/elasticsearch
sudo chown elasticsearch:elasticsearch /var/backups/elasticsearch
sudo systemctl restart elasticsearch

Register the repository:

curl -k -u elastic:PASS -X PUT "https://localhost:9200/_snapshot/backups" \
  -H 'Content-Type: application/json' -d '{
    "type": "fs",
    "settings": { "location": "/var/backups/elasticsearch", "compress": true }
  }'

S3 Repository (Recommended for Production)

Install the S3 plugin:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3

Store credentials in the keystore:

echo "YOUR_S3_ACCESS_KEY" | sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add -x s3.client.default.access_key
echo "YOUR_S3_SECRET_KEY" | sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add -x s3.client.default.secret_key
sudo systemctl restart elasticsearch

Register:

curl -k -u elastic:PASS -X PUT "https://localhost:9200/_snapshot/s3-backups" \
  -H 'Content-Type: application/json' -d '{
    "type": "s3",
    "settings": {
      "bucket": "my-elk-snapshots",
      "endpoint": "s3.eu-central-1.amazonaws.com",
      "protocol": "https",
      "base_path": "elk-prod/"
    }
  }'

Snapshot Lifecycle Management (SLM)

Schedule daily snapshots that retain the last 30 days:

curl -k -u elastic:PASS -X PUT "https://localhost:9200/_slm/policy/daily-snapshots" \
  -H 'Content-Type: application/json' -d '{
    "schedule": "0 30 1   ?",
    "name": "<daily-snap-{now/d}>",
    "repository": "s3-backups",
    "config": { "indices": ["logs-","metrics-",".kibana*"], "include_global_state": true },
    "retention": { "expire_after": "30d", "min_count": 7, "max_count": 60 }
  }'

Restore with:

curl -k -u elastic:PASS -X POST "https://localhost:9200/_snapshot/s3-backups/daily-snap-2026.04.15/_restore"

Troubleshooting

Viewing Logs

sudo journalctl -u elasticsearch -f
sudo journalctl -u kibana -f
sudo journalctl -u logstash -f
sudo tail -f /var/log/elasticsearch/elk-prod.log

Health Checks

# Cluster health
curl -k -u elastic:PASS https://localhost:9200/_cluster/health?pretty
Disk usage per node
curl -k -u elastic:PASS https://localhost:9200/_cat/allocation?v
Index sizes
curl -k -u elastic:PASS https://localhost:9200/_cat/indices?v&s=store.size:desc

FAQ

ELK vs Grafana Loki — which should I pick?

Loki (from Grafana Labs) indexes only log labels, not log content, which makes it dramatically cheaper in storage but limits the richness of ad-hoc queries. It pairs naturally with Grafana dashboards and Prometheus metrics, fitting a "metrics first, logs as supporting evidence" philosophy. Choose Loki when your log volumes are high (terabytes per day), you already run Grafana/Prometheus, and your queries are predictable and label-filterable.

ELK indexes every field, which makes arbitrary post-hoc queries, complex aggregations, and full-text search trivial. It also gives you the SIEM app, APM, ML, and alerting in one bundle. Choose ELK when you need security analytics, compliance-grade search, or rich free-form queries across structured logs.

Many mature teams run both — Loki for cheap high-volume log storage, ELK for security and compliance logs that need retention and search.

ELK vs Graylog?

Graylog is an older open-source log management platform that uses Elasticsearch/OpenSearch under the hood and adds MongoDB for its own metadata. It has a cleaner UI for log-centric workflows, stream-based routing, and role-based search restrictions that used to be Enterprise-tier in ELK. The trade-off is that Graylog does not offer the APM, SIEM, ML, or vector-search features Kibana has in 8.x. For pure log management, Graylog is still excellent; for observability + security in one stack, ELK wins.

ELK vs OpenSearch?

OpenSearch is AWS's Apache-2.0 fork of Elasticsearch 7.10, maintained by the Linux Foundation. Functionally it is a drop-in replacement for basic log search, and all AWS services (Logstash-compatible ingestion, OpenSearch Dashboards as a Kibana fork) work. However, since the 2021 fork, Elastic has added significant features OpenSearch does not have: ES|QL (piped query language), native vector search optimisations, the Security/SIEM detection rules, Fleet-managed Elastic Agent, and much more advanced ML. OpenSearch's ML plugin catches up on some features but lags behind. Choose OpenSearch only if your legal/procurement team refuses SSPL; otherwise Elastic 8.x Basic gives you more for free.

What is the minimum VPS sizing for ELK?

The floor is 4 vCPU / 8 GB RAM / 40 GB SSD for a hobby/dev install ingesting a few hundred MB/day. Realistic production starts at 6 vCPU / 16 GB RAM / 200 GB NVMe (our CloudCore Business plan), which comfortably handles 20-50 GB/day with 30-day retention. Multi-tenant or high-cardinality workloads scale to dedicated 32-64 GB nodes. Disk I/O matters as much as RAM — NVMe is non-negotiable for Elasticsearch.

Cloud ELK vs self-hosted — what is the break-even?

Elastic Cloud starts around $95/month for a 2 GB RAM hot tier node and climbs quickly once you add ingest volume, retention, or ML nodes. A comparable self-hosted VPS at €35.99/month breaks even within the first billing cycle and gives you better hardware (more RAM, NVMe). Break-even flips in Elastic Cloud's favour only when your ops time exceeds ~2 hours/month or when you truly need multi-region replication. For most single-team deployments under 100 GB/day, self-hosted is meaningfully cheaper.

How long should I retain logs?

Retention is driven by three factors: compliance, operational need, and disk budget.

• 7 days hot — enough for on-call debugging and same-week incident forensics.
• 30 days warm — covers monthly audits and recurring incident patterns.
• 90 days cold — required minimum for PCI-DSS and many internal compliance policies.
• 1 year archive — common for SOC2 audit trails and security investigations. Keep these in S3 as searchable snapshots, not on hot disk.

Do I need Logstash, or can I go Filebeat → Elasticsearch directly?

For simple log shipping with minor enrichment, Filebeat modules + Elasticsearch ingest pipelines are sufficient and have lower overhead. Skip Logstash in that case. You need Logstash when you have:

• Complex multi-stage enrichment (multiple grok passes, database lookups, HTTP calls)
• Multiple outputs (ES + S3 + Kafka + Splunk simultaneously)
• A persistent queue for reliability when ES is unavailable
• Non-Elastic input sources (JDBC, Kafka, JMS)

How do I scale from single-node to a cluster?

Elasticsearch then auto-distributes shards across nodes and survives single-node failures transparently.

Next Steps

Your ELK Stack is running, ingesting, and alerting. Common follow-ups:

• Ship Docker container logs -- Follow our Docker install guide and enable the Filebeat docker module. Container metadata (image, labels, compose project) auto-tags every event.

• Metrics with Metricbeat -- Install Metricbeat alongside Filebeat to collect system, Nginx, PostgreSQL, and Docker metrics into the same ELK cluster. Kibana's Metrics app gives you host/container-level views comparable to a lightweight Prometheus + Grafana setup.

• Pair ELK with Loki for cheap archive -- For ultra-high-volume logs that do not need field-level search, install Grafana Loki in parallel. Keep 7 days of full-fidelity logs in Elasticsearch and 90 days of label-indexed logs in Loki at a fraction of the storage cost.

• Automate index templates with Terraform -- Use the Elasticstack Terraform provider to version-control ILM policies, index templates, roles, and Watcher rules alongside the rest of your infrastructure.

• Enable APM -- The Elastic APM server ships inside the Kibana Integrations UI. Instrument your Node.js, Python, Java, or Go services and get distributed tracing + service maps in the same Kibana UI as your logs.

• Install Elastic Agent + Fleet -- Fleet centralises Beats management. Instead of editing filebeat.yml on 50 hosts, you configure integrations in Kibana and Elastic Agent pulls its own config.

Skip the Manual Install — Get ELK Stack Pre-Installed

Our Observability-Ready VPS comes with Elasticsearch, Kibana, Logstash, and Filebeat pre-configured with TLS, Nginx, ILM, and Kibana dashboards. Deploy in five minutes and start ingesting logs immediately.

- Elasticsearch 8.x with auto-generated TLS and elastic password

- Kibana behind Nginx with Let's Encrypt certificate

- Logstash pipeline wired to Beats input on 5044

- Filebeat agent ready to enrol additional edge hosts

- ILM policy and snapshot repository configured

- systemd services with production-tuned heap and limits

Deploy Your ELK VPS Now — CloudCore Business starts at EUR 35.99/month.