How to Install etcd on Ubuntu 24.04 VPS: Distributed Key-Value Store for Coordination
etcd is the quiet workhorse behind some of the most important infrastructure on the internet. It stores the entire desired state of every Kubernetes cluster, drives service discovery in CoreDNS, holds leader election locks in projects like Patroni and Vitess, and backs configuration distribution for countless smaller systems. This guide walks you through installing etcd v3.5 on Ubuntu 24.04 from a first single-node developer setup to a hardened 3-node production cluster with TLS, authentication, backups, and Prometheus metrics.
Skip the manual setup? Deploy a 3-node etcd cluster on our CloudCore Professional VPS plans with cloud-init automation in under 10 minutes per node.
Table of Contents
What is etcd?
etcd is a strongly consistent, distributed key-value store written in Go. It was created at CoreOS in 2013, now lives under the Cloud Native Computing Foundation, and uses the Raft consensus algorithm to replicate every write across a cluster of nodes. Every client sees the same data in the same order — etcd is the textbook example of a linearizable store.
The data model is intentionally minimal: keys are byte strings, values are byte strings, and the keyspace is flat but navigable by prefix. On top of that small core, etcd provides a handful of powerful primitives: atomic compare-and-swap transactions, leases that auto-expire keys when a client disappears, watches that stream every mutation on a prefix, and a built-in auth system with users and roles.
etcd is most famous as the backing store for Kubernetes — every Pod, Service, ConfigMap, and Secret you create is a serialized object under a key in etcd. But it's equally at home outside Kubernetes. Patroni uses it to elect the primary PostgreSQL replica. CoreDNS uses it as a dynamic record source. Traefik reads its configuration from etcd. M3DB uses it for cluster topology. Smaller projects use it as a feature flag store, a distributed lock service, or a shared configuration bus between microservices.
Why Self-Host etcd on Your VPS?
Hosted coordination services like Zookeeper-as-a-service exist, but running your own etcd cluster is often the better call:
- Low, predictable latency — Raft commits are bottlenecked on disk fsync and network RTT. A 3-node cluster on VPS instances in the same region commits in 1–3 ms. Managed services typically add 10–30 ms of hop overhead.
- Full protocol access — Every etcd feature is available: gRPC API, watch streams, lease grants, custom auth roles, and snapshot tooling. No per-operation pricing or feature gates.
- Own your data — etcd often holds the only authoritative copy of leader election state or service registration. Keeping that on infrastructure you control is safer than trusting a third party.
- Pair it with Kubernetes — Whether you're running k3s, k0s, or kubeadm, you can either let the distribution manage its own etcd or run an external etcd cluster that you operate independently. External etcd makes it easier to upgrade Kubernetes without disturbing state.
- Easy Prometheus integration — etcd exports Raft, disk, and gRPC metrics natively on
/metrics. Drop it into your existing Prometheus + Grafana stack with zero glue code.
Standalone etcd vs. Kubernetes etcd
| Use Case | Deployment | Operated by |
|---|---|---|
| Kubernetes control plane (kubeadm default) | Static pod, one per master | kubeadm, lifecycle tied to kubelet |
| Kubernetes control plane (external) | Dedicated VMs running etcd | You (or this guide) |
| Service discovery, feature flags, config | Standalone 3-node cluster | You (this guide) |
| Distributed locks, leader election | Standalone 3-node cluster | You (this guide) |
| Single-app coordination, dev | Single-node on one VPS | You (Step 3) |
Prerequisites
- One or three VPS instances running Ubuntu 24.04 LTS with root or sudo access
- SSH access to each server
- Low-latency network between nodes — for a 3-node cluster, run all nodes in the same region (ideally same datacenter)
- At least 2 GB RAM and 10 GB disk per node for modest workloads; etcd is light but writes are fsync-bound so NVMe is strongly preferred
- Ports open between nodes:
2379/tcp(client) and2380/tcp(peer)
Recommended Plan: CloudCore Professional>
For a production-ready 3-node etcd cluster, we recommend three CloudCore Professional instances:>
- 6 vCPU cores
- 12 GB RAM
- 100 GB NVMe SSD (essential for Raft commit latency)
- Unmetered bandwidth
- EUR 19.99/month per node>
NVMe storage is the single biggest factor in etcd write throughput — the disk fsync for the Raft log is on the hot path of every write.
Connect to your first server:
ssh root@your-server-ipStep 1: Update System and Create User
Refresh your package index and install a handful of helpers used later in the guide.
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl tar ca-certificates gnupgCreate a dedicated unprivileged system user for etcd so the daemon never runs as root:
sudo useradd --system --home /var/lib/etcd --shell /sbin/nologin --comment "etcd service" etcd
sudo mkdir -p /var/lib/etcd /etc/etcd
sudo chown -R etcd:etcd /var/lib/etcd /etc/etcd
sudo chmod 700 /var/lib/etcdThe 700 permission on the data directory is important — etcd will refuse to start if the directory is world-readable, because it may contain sensitive secret values.
Step 2: Download the etcd v3.5 Binary
etcd ships static Linux binaries on its GitHub releases page. Download the latest v3.5 tarball directly.
ETCD_VER=v3.5.17
ARCH=$(dpkg --print-architecture) # amd64 on most VPS, arm64 on ARM
cd /tmp
curl -L -o etcd.tar.gz \
https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-${ARCH}.tar.gz
tar xzf etcd.tar.gzInstall the two binaries into /usr/local/bin:
sudo install -m 0755 /tmp/etcd-${ETCD_VER}-linux-${ARCH}/etcd /usr/local/bin/etcd
sudo install -m 0755 /tmp/etcd-${ETCD_VER}-linux-${ARCH}/etcdctl /usr/local/bin/etcdctl
sudo install -m 0755 /tmp/etcd-${ETCD_VER}-linux-${ARCH}/etcdutl /usr/local/bin/etcdutlVerify:
etcd --version
etcdctl versionExpected output:
etcd Version: 3.5.17 Git SHA: ... Go Version: go1.22.8 Go OS/Arch: linux/amd64
etcdctl version: 3.5.17 API version: 3.5
Always use the v3 API — ETCDCTL_API=3 is the default in 3.5.x but export it anyway so shell history is explicit:
echo 'export ETCDCTL_API=3' | sudo tee /etc/profile.d/etcd.sh
source /etc/profile.d/etcd.shStep 3: Run a Single-Node Dev Instance
For local experimentation on one server, the simplest viable etcd config is a single-node cluster listening on localhost. This is only appropriate for development — a single node has no redundancy.
Create /etc/etcd/etcd.conf.yml:
sudo tee /etc/etcd/etcd.conf.yml > /dev/null <<'EOF'
name: dev-node
data-dir: /var/lib/etcd
listen-client-urls: http://127.0.0.1:2379
advertise-client-urls: http://127.0.0.1:2379
listen-peer-urls: http://127.0.0.1:2380
initial-advertise-peer-urls: http://127.0.0.1:2380
initial-cluster: dev-node=http://127.0.0.1:2380
initial-cluster-state: new
initial-cluster-token: dev-cluster-1
enable-v2: false
EOF
sudo chown etcd:etcd /etc/etcd/etcd.conf.ymlCreate a systemd unit at /etc/systemd/system/etcd.service:
sudo tee /etc/systemd/system/etcd.service > /dev/null <<'EOF' [Unit] Description=etcd key-value store Documentation=https://etcd.io/docs/ After=network-online.target Wants=network-online.target[Service] Type=notify User=etcd Group=etcd ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.conf.yml Restart=on-failure RestartSec=5s LimitNOFILE=65536
[Install] WantedBy=multi-user.target EOF
sudo systemctl daemon-reload sudo systemctl enable --now etcd sudo systemctl status etcd --no-pager
Expected status:
● etcd.service - etcd key-value store
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-04-16 12:00:00 UTC; 3s ago
Main PID: 4321 (etcd)Sanity check the cluster:
etcdctl endpoint health
etcdctl endpoint status --write-out=tableExpected output:
127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.234ms
+----------------+------------------+---------+---------+-----------+... | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | +----------------+------------------+---------+---------+-----------+ | 127.0.0.1:2379 | 8e9e05c52164694d | 3.5.17 | 20 kB | true | +----------------+------------------+---------+---------+-----------+
You now have a working dev etcd. Before moving on, stop it so we can reconfigure for clustering:
sudo systemctl stop etcd
sudo rm -rf /var/lib/etcd/*Step 4: Bootstrap a 3-Node Cluster
For production, run an odd-numbered cluster. Three nodes tolerate the loss of one; five tolerate two. The write-latency cost of going above five is usually not worth it unless you have a strong availability requirement.
Assume three VPS instances with these private IPs (substitute your own):
| Node | Hostname | IP |
|---|---|---|
| 1 | etcd1 | 10.0.0.11 |
| 2 | etcd2 | 10.0.0.12 |
| 3 | etcd3 | 10.0.0.13 |
sudo ufw allow from 10.0.0.0/24 to any port 2379 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 2380 proto tcpOn each node, create /etc/etcd/etcd.conf.yml with the node-specific name and IP, but the same initial-cluster string. Node 1 looks like this:
name: etcd1 data-dir: /var/lib/etcdlisten-client-urls: http://10.0.0.11:2379,http://127.0.0.1:2379 advertise-client-urls: http://10.0.0.11:2379
listen-peer-urls: http://10.0.0.11:2380 initial-advertise-peer-urls: http://10.0.0.11:2380
initial-cluster: etcd1=http://10.0.0.11:2380,etcd2=http://10.0.0.12:2380,etcd3=http://10.0.0.13:2380 initial-cluster-state: new initial-cluster-token: prod-cluster-1
enable-v2: false logger: zap log-level: info
Node 2 uses name: etcd2 and swaps 10.0.0.11 for 10.0.0.12; Node 3 uses 10.0.0.13. The initial-cluster list is identical on every node.
Understanding the four URL settings is the most common point of confusion:
listen-client-urls— the addresses etcd binds to for application/etcdctl connections. Include127.0.0.1for local tooling.advertise-client-urls— the addresses etcd tells clients to use. Must be reachable from clients. Do not advertise127.0.0.1.listen-peer-urls— the addresses etcd binds to for Raft traffic from other members.initial-advertise-peer-urls— the addresses etcd tells other members to contact it on. Must match what other members see ininitial-cluster.
# On each of the three nodes:
sudo systemctl daemon-reload
sudo systemctl enable --now etcdCheck cluster membership from any node:
export ENDPOINTS=10.0.0.11:2379,10.0.0.12:2379,10.0.0.13:2379
etcdctl --endpoints=$ENDPOINTS member list --write-out=table
etcdctl --endpoints=$ENDPOINTS endpoint status --write-out=tableExpected output:
+------------------+---------+-------+----------------------+----------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+----------------------+----------------------+
| 8211f1d0f64f3269 | started | etcd1 | http://10.0.0.11:2380 | http://10.0.0.11:2379 |
| 91bc3c398fb3c146 | started | etcd2 | http://10.0.0.12:2380 | http://10.0.0.12:2379 |
| fd422379fda50e48 | started | etcd3 | http://10.0.0.13:2380 | http://10.0.0.13:2379 |
+------------------+---------+-------+----------------------+----------------------+Exactly one node will show IS LEADER = true in endpoint status. Leadership rotates automatically if that node fails.
Step 5: Enable TLS for Peer and Client Traffic
Running etcd in cleartext is acceptable only on a trusted private network. For production, use mTLS on both the peer port (2380) and the client port (2379). You can use separate CAs for peer and client traffic — we'll generate both with cfssl.
Install cfssl on the node you'll use as the certificate issuer (any workstation or one of the etcd nodes):
sudo apt install -y golang-cfsslCreate the CAs
mkdir -p ~/etcd-ca && cd ~/etcd-cacat > ca-config.json <<'EOF' { "signing": { "default": { "expiry": "87600h" }, "profiles": { "peer": { "usages": ["signing","key encipherment","server auth","client auth"], "expiry": "87600h" }, "client": { "usages": ["signing","key encipherment","client auth"], "expiry": "87600h" }, "server": { "usages": ["signing","key encipherment","server auth"], "expiry": "87600h" } } } } EOF
cat > ca-csr.json <<'EOF' { "CN": "etcd-ca", "key": {"algo":"rsa","size":4096}, "names": [{"O":"vps-server.host","OU":"etcd"}] } EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
You now have ca.pem (the trust root) and ca-key.pem (keep secret).
Issue node certificates
For each node, generate a certificate valid for its IP and hostname:
for NODE in etcd1:10.0.0.11 etcd2:10.0.0.12 etcd3:10.0.0.13; do
NAME=${NODE%:}; IP=${NODE#:}
cat > ${NAME}-csr.json <<EOF
{ "CN": "${NAME}", "hosts": ["${NAME}","${IP}","127.0.0.1","localhost"],
"key": {"algo":"rsa","size":2048},
"names": [{"O":"vps-server.host"}] }
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=peer ${NAME}-csr.json | cfssljson -bare ${NAME}
doneIssue a client certificate for etcdctl
cat > client-csr.json <<'EOF'
{ "CN": "root", "hosts": [""], "key": {"algo":"rsa","size":2048},
"names": [{"O":"vps-server.host"}] }
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=client client-csr.json | cfssljson -bare clientDistribute and switch etcd to TLS
Copy the CA cert plus that node's key pair into /etc/etcd/certs/ on each node:
sudo mkdir -p /etc/etcd/certs
sudo cp ca.pem etcd1.pem etcd1-key.pem /etc/etcd/certs/ # adjust per node
sudo chown -R etcd:etcd /etc/etcd/certs
sudo chmod 600 /etc/etcd/certs/*-key.pemUpdate each node's /etc/etcd/etcd.conf.yml — change http:// to https:// on every URL and add the TLS blocks:
listen-client-urls: https://10.0.0.11:2379,https://127.0.0.1:2379 advertise-client-urls: https://10.0.0.11:2379 listen-peer-urls: https://10.0.0.11:2380 initial-advertise-peer-urls: https://10.0.0.11:2380 initial-cluster: etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.12:2380,etcd3=https://10.0.0.13:2380client-transport-security: cert-file: /etc/etcd/certs/etcd1.pem key-file: /etc/etcd/certs/etcd1-key.pem trusted-ca-file: /etc/etcd/certs/ca.pem client-cert-auth: true
peer-transport-security: cert-file: /etc/etcd/certs/etcd1.pem key-file: /etc/etcd/certs/etcd1-key.pem trusted-ca-file: /etc/etcd/certs/ca.pem peer-client-cert-auth: true
Restart the cluster node by node:
sudo systemctl restart etcdVerify with etcdctl using the client certificate:
etcdctl \
--endpoints=https://10.0.0.11:2379 \
--cacert=/etc/etcd/certs/ca.pem \
--cert=client.pem --key=client-key.pem \
endpoint healthYou'll type those flags a lot. Put them in your shell profile:
export ETCDCTL_ENDPOINTS=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379
export ETCDCTL_CACERT=/etc/etcd/certs/ca.pem
export ETCDCTL_CERT=/etc/etcd/certs/client.pem
export ETCDCTL_KEY=/etc/etcd/certs/client-key.pemStep 6: Use etcdctl — put, get, watch, leases, transactions
With TLS in place, exercise the main API features.
Basic put and get
etcdctl put /config/app/theme "dark"
etcdctl get /config/app/themeOutput:
/config/app/theme
darkList all keys under a prefix:
etcdctl put /config/app/locale "en_US"
etcdctl get --prefix /config/app/Delete a key or a prefix:
etcdctl del /config/app/theme
etcdctl del --prefix /config/app/Watches
A watch streams every change to a key or prefix. In terminal 1:
etcdctl watch --prefix /config/app/In terminal 2:
etcdctl put /config/app/theme "light"
etcdctl put /config/app/version "2.1"Terminal 1 prints:
PUT
/config/app/theme
light
PUT
/config/app/version
2.1Watches are how Kubernetes controllers, CoreDNS, and Traefik react to configuration changes in real time without polling.
Leases (TTL keys)
A lease is a cluster-tracked TTL you can attach to any key. When the lease expires or its owner crashes without renewing it, all keys bound to it are deleted. This is the basis of distributed locks and service registration.
LEASE=$(etcdctl lease grant 30 | awk '{print $2}')
echo "Lease: $LEASE"etcdctl put /services/web/node-1 "10.0.0.50:8080" --lease=$LEASE
etcdctl get /services/web/node-1
Keep it alive in another terminal:
etcdctl lease keep-alive $LEASEStop the keep-alive process, wait 30 seconds, and the key vanishes automatically.
Transactions (compare-and-swap)
Transactions let you make writes conditional on existing state — the building block of distributed locks and optimistic concurrency.
etcdctl txn <<'EOF' mod("/leaders/db") = "0"put /leaders/db "node-1"
put /leaders/db "lost-race" EOF
The syntax is: compare clause(s) -> empty line -> success ops -> empty line -> failure ops. mod("key") = "0" means "only if this key has never been written". The example takes leadership atomically — the first node to run it wins, subsequent runs fall through to the failure branch.
Step 7: Enable Authentication (Users and Roles)
Even with mTLS, turning on etcd's built-in auth gives you per-user RBAC over the keyspace.
Create the mandatory root user first — without it, auth cannot be enabled.
etcdctl user add root
Enter a strong password when prompted
etcdctl user grant-role root rootCreate an application role limited to a prefix:
etcdctl role add app-reader etcdctl role grant-permission app-reader --prefix=true read /config/app/
etcdctl role add app-writer etcdctl role grant-permission app-writer --prefix=true readwrite /config/app/
Create a user and bind it to a role:
etcdctl user add app-svc
etcdctl user grant-role app-svc app-writerTurn auth on:
etcdctl auth enableAfter this point every command must carry credentials:
etcdctl --user=app-svc:password put /config/app/feature-x "on"
etcdctl --user=app-svc:password get /config/app/feature-xAttempts outside the granted prefix fail:
etcdctl --user=app-svc:password put /secrets/db-password "hunter2"
Error: etcdserver: permission denied
To manage the cluster itself, always authenticate as root:
etcdctl --user=root:password member listStep 8: Snapshots — Save and Restore
Even a 3-node cluster needs off-box backups. Disk corruption, a misconfigured apt upgrade, or a bad etcdctl del --prefix / all affect every replica simultaneously.
Take a snapshot
A snapshot is a single file containing a consistent point-in-time copy of the entire keyspace.
sudo -u etcd etcdctl \
--endpoints=https://10.0.0.11:2379 \
snapshot save /var/lib/etcd/snapshots/etcd-$(date +%F-%H%M).dbVerify it:
etcdutl snapshot status /var/lib/etcd/snapshots/etcd-2026-04-16-1200.db --write-out=tableExpected:
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 7d2bfd1a | 142 | 512 | 2.1 MB |
+----------+----------+------------+------------+Schedule it with a systemd timer:
sudo tee /etc/systemd/system/etcd-snapshot.service > /dev/null <<'EOF' [Unit] Description=etcd snapshot [Service] Type=oneshot User=etcd Environment=ETCDCTL_API=3 ExecStart=/usr/local/bin/etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/etcd/certs/ca.pem \ --cert=/etc/etcd/certs/client.pem \ --key=/etc/etcd/certs/client-key.pem \ snapshot save /var/lib/etcd/snapshots/etcd-%i.db EOFsudo tee /etc/systemd/system/etcd-snapshot.timer > /dev/null <<'EOF' [Unit] Description=Hourly etcd snapshot [Timer] OnCalendar=hourly Persistent=true [Install] WantedBy=timers.target EOF
sudo mkdir -p /var/lib/etcd/snapshots sudo chown etcd:etcd /var/lib/etcd/snapshots sudo systemctl enable --now etcd-snapshot.timer
Then copy snapshots off-server with restic, rclone, or a plain aws s3 cp hook.
Restore a cluster from a snapshot
Restore is a destructive operation — you rebuild the cluster from the snapshot as if bootstrapping for the first time.
On each node:
sudo systemctl stop etcd sudo mv /var/lib/etcd/member /var/lib/etcd/member.bak
sudo -u etcd etcdutl snapshot restore /var/lib/etcd/snapshots/etcd-2026-04-16-1200.db \ --name etcd1 \ --initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.12:2380,etcd3=https://10.0.0.13:2380 \ --initial-cluster-token prod-cluster-restore-1 \ --initial-advertise-peer-urls https://10.0.0.11:2380 \ --data-dir /var/lib/etcd
Adjust --name and the advertise URL per node. Start all three nodes again — they'll re-form a cluster using the restored data with a fresh cluster token, preventing accidental merging with any old members still running.
Step 9: Prometheus Metrics
etcd exposes rich Prometheus metrics on the client port under /metrics. Scrape it with mTLS from Prometheus:
# prometheus.yml
scrape_configs:
- job_name: etcd
scheme: https
tls_config:
ca_file: /etc/prometheus/etcd-ca.pem
cert_file: /etc/prometheus/etcd-client.pem
key_file: /etc/prometheus/etcd-client-key.pem
static_configs:
- targets:
- 10.0.0.11:2379
- 10.0.0.12:2379
- 10.0.0.13:2379Key metrics to alert on:
| Metric | Meaning | Alert threshold |
|---|---|---|
etcd_server_has_leader | 1 if the node sees a leader | == 0 for 1m |
etcd_server_leader_changes_seen_total | Count of leader elections | rate > 3 per 10m = unstable network |
etcd_disk_wal_fsync_duration_seconds | WAL fsync latency | p99 > 10 ms = slow disk |
etcd_disk_backend_commit_duration_seconds | Backend commit latency | p99 > 25 ms |
etcd_server_proposals_failed_total | Failed Raft proposals | rate > 0 = capacity problem |
etcd_mvcc_db_total_size_in_bytes | On-disk DB size | approaching --quota-backend-bytes |
# No leader for 1 minute
sum(etcd_server_has_leader) by (job) < 2Disk fsync p99 above 10ms
histogram_quantile(0.99,
rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
) > 0.01The official Grafana dashboard ID is 3070; import it for instant visualization of every metric above.
Performance Tuning
etcd's throughput is bottlenecked on two things: disk fsync and network RTT between members.
Disk
Use NVMe. Rotational disks cannot keep up with Raft WAL fsyncs under load. If you're on a cloud VPS, check that your volume is not an aggregated network-attached disk with variable latency. Run fio with small random writes to verify <5 ms p99 fsync.
Raise the backend quota if your dataset is large (default 2 GB):
# etcd.conf.yml
quota-backend-bytes: 8589934592 # 8 GBCompaction and defragmentation
Every write creates a new revision. Without compaction, old revisions accumulate and eat disk.
# Auto-compact every hour, keeping 1h of history:
auto-compaction-mode: periodic
auto-compaction-retention: "1h"Defragment periodically (rewrites the boltdb backend):
etcdctl defrag --clusterRun this during low-traffic periods — it briefly blocks writes on the defragmenting member.
Heartbeat and election timeouts
For high-latency links (e.g. WAN members), tune:
heartbeat-interval: 100 # ms, default 100
election-timeout: 1000 # ms, default 1000Rule of thumb: election timeout must be at least 10x heartbeat and well above your 99th percentile RTT.
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
etcdserver: mvcc: database space exceeded | DB hit quota-backend-bytes | Run etcdctl compact + etcdctl defrag --cluster, raise the quota |
etcdserver: request timed out | Slow disk fsync or leader flap | Check etcd_disk_wal_fsync_duration_seconds, move to NVMe |
rpc error: code = Unavailable desc = transport is closing | Cert expired or clock skew | Re-issue certs, run timedatectl to confirm NTP sync |
| Nodes cannot form cluster at bootstrap | initial-cluster differs between nodes | Every node must have the identical initial-cluster list |
tls: bad certificate in peer logs | Mismatched peer CA or SAN does not include the IP | Regenerate peer cert with hostname and IP in hosts |
| High leader changes | Network partition or CPU starvation | Check loss/jitter between members; pin etcd CPU priority |
etcdserver: permission denied after enabling auth | Missing --user= flag or insufficient role | Use etcdctl --user=root:pw for admin, grant prefix on role |
Error: context deadline exceeded from etcdctl | Wrong endpoint or firewall blocking 2379 | Confirm ufw allow, test with nc -vz 10.0.0.11 2379 |
Useful log commands
sudo journalctl -u etcd -f
sudo journalctl -u etcd --since "10 min ago" | grep -i warnFAQ
How many nodes should an etcd cluster have?
Always an odd number. A 3-node cluster tolerates 1 failure, a 5-node cluster tolerates 2, a 7-node cluster tolerates 3. Above 7 the Raft write latency penalty (every write must reach a majority) usually outweighs the extra availability, because Raft commits are serialized through the leader. 3 or 5 is right for 99% of deployments.
Can I run etcd on a single node in production?
Only for stateless or easily-rebuilt workloads where losing etcd is an inconvenience, not a disaster. A single node has no redundancy — one disk corruption event and the keyspace is gone. Use 3 nodes minimum for anything you cannot reconstruct from another source of truth.
What's the difference between the peer port and the client port?
Port 2380 carries Raft traffic between etcd members (AppendEntries, heartbeats, leader elections). Port 2379 accepts requests from applications and etcdctl. In a hardened deployment you can use different CAs for each — so leaking a client cert doesn't let an attacker impersonate a Raft member.
Does Kubernetes use the same etcd binary?
Yes. kubeadm, k3s (by default), k0s, and every other major Kubernetes distribution ship the upstream etcd v3 binary from github.com/etcd-io/etcd. The wire format, API, and snapshot format are identical, which is why a Kubernetes admin who can operate etcd for kube can also operate a standalone etcd for a Patroni cluster or a Traefik config store.
How do I back up etcd safely?
Run etcdctl snapshot save on any healthy member — the result is a consistent point-in-time file. Schedule it hourly via systemd timer (Step 8) and ship the file off-box to S3, a second VPS, or cold storage. Verify snapshots with etcdutl snapshot status so you don't discover they're corrupt during a real restore.
Can I add or remove nodes from a running cluster?
Yes — use etcdctl member add and etcdctl member remove. Add a fresh node with --initial-cluster-state existing in its config so it catches up from the leader instead of trying to bootstrap a new cluster. Remove a failed node before replacing it, otherwise the cluster may lose quorum mid-change.
What should I monitor first?
etcd_server_has_leader (alert when zero), WAL fsync p99 (alert above 10 ms), and etcd_mvcc_db_total_size_in_bytes (alert at 80% of quota). Those three catch nearly every practical etcd incident before it turns into an outage.
Next Steps
Now that etcd is running, here are natural follow-ups:
- Install k3s on Ubuntu — Point k3s at your external etcd with
--datastore-endpointto separate control-plane state from worker nodes. This is the recommended pattern for larger k3s clusters. - Install Consul on Ubuntu — Consul is the other common choice for service discovery and KV. Compare it side by side with etcd: Consul has richer service-mesh features while etcd wins on raw write throughput and simplicity.
- Install Prometheus on Ubuntu — Finish the monitoring loop by scraping the etcd
/metricsendpoint shown in Step 9. Import Grafana dashboard 3070 for an instant operational view. - Add Patroni for HA PostgreSQL — Patroni uses etcd as its DCS to elect and fence a PostgreSQL primary. Your 3-node etcd cluster can back a 3-node Postgres HA setup.
- Read the upstream docs — The official etcd documentation has deep operational guides, the full gRPC API reference, and migration notes between minor versions.
Skip the Manual Setup — Get etcd Pre-Configured>
Our CloudCore Professional VPS plans ship with cloud-init templates for etcd v3.5 clusters. Deploy three nodes with TLS, systemd units, snapshot timers, and Prometheus metrics already wired in.>
- 3-node cluster bootstrap in under 10 minutes
- mTLS generated and distributed automatically
- Hourly snapshots shipped to object storage
- Grafana dashboard 3070 preloaded>
Deploy Your etcd Cluster — Plans start at EUR 19.99/month per node.