How to Install Apache Kafka on Ubuntu 24.04 VPS: Distributed Event Streaming Platform

Apache Kafka is the de-facto standard for high-throughput event streaming, powering pipelines at Netflix, LinkedIn, Uber, and thousands of engineering teams that need to move millions of events per second between services. Running Kafka on your own Ubuntu 24.04 VPS gives you a durable, replayable log for microservices communication, analytics ingestion, change data capture, and real-time applications -- without paying per-partition or per-GB fees to a managed SaaS. This guide walks through a full production install using Kafka 3.7+ with KRaft (the ZooKeeper-free successor), systemd supervision, SASL_SSL authentication, a web UI, and a three-node cluster topology.

Planning for production workloads? Our CloudCore Professional plan gives you the 12 GB RAM and NVMe storage needed for a healthy Kafka broker, with the option to scale to three nodes for full replication.

Table of Contents

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform originally built at LinkedIn and donated to the Apache Software Foundation in 2011. At its core, Kafka is a durable, append-only commit log partitioned across a cluster of servers (brokers). Producers write events to named streams called topics; consumers read events from those topics independently, at their own pace, and can replay history by resetting their offset. Because every event is persisted to disk and replicated across brokers, Kafka functions simultaneously as a message queue, an event bus, and a storage system.

Kafka's design makes it uniquely suited to a handful of workloads where traditional message brokers fall short. Event-driven microservices use Kafka as the backbone connecting services so that each team can publish domain events without knowing which services consume them. Real-time analytics pipelines stream clickstream, IoT, and telemetry data into Kafka topics, then fan out to systems like ClickHouse, Druid, Pinot, or a data warehouse. Change data capture (CDC) tools like Debezium tail MySQL, PostgreSQL, and MongoDB binlogs into Kafka, giving downstream systems a live replica of every database write. Log aggregation pipelines collect application logs from thousands of servers into a single topic for indexing. Stream processing frameworks like Kafka Streams, Apache Flink, and ksqlDB operate directly on Kafka topics to compute joins, aggregations, and windowed analytics.

The 3.x release line introduced the biggest architectural change in Kafka's history: KRaft mode (Kafka Raft Metadata mode). Earlier versions required a separate ZooKeeper ensemble to store cluster metadata, adding operational complexity and a second distributed system to keep healthy. KRaft replaces ZooKeeper with a built-in Raft consensus protocol running inside Kafka itself. Starting with Kafka 3.3 KRaft was marked production-ready, and from 3.5 onward it became the default mode for new clusters. Kafka 4.0 (scheduled for 2025) removes ZooKeeper entirely. This guide uses Kafka 3.7+ in KRaft mode exclusively -- there is no reason to deploy a new cluster any other way.

Why Self-Host Kafka Instead of Using Confluent Cloud?

Managed Kafka services like Confluent Cloud, Amazon MSK, and Aiven are excellent, but they come with trade-offs that push many teams toward a self-hosted deployment on their own VPS.

• Predictable, flat-rate cost -- Confluent Cloud bills for ingress, egress, storage, and partitions. A workload that looks cheap at 1 MB/s can balloon to hundreds or thousands of dollars per month once traffic grows. A single CloudCore Professional VPS handles tens of MB/s on a flat EUR 19.99/month bill, and scaling is linear and transparent.
• No egress charges -- Managed Kafka on AWS or GCP charges for every byte leaving the service. Self-hosting on a VPS with unmetered bandwidth eliminates that line item entirely.
• Full protocol access -- You get every Kafka feature on day one: tiered storage, compacted topics, transactional producers, exactly-once semantics, and the full admin API. Some managed services gate features behind higher-priced tiers.
• Complete data sovereignty -- Event streams often carry PII, financial transactions, or proprietary business data. Self-hosting lets you choose the jurisdiction, apply your own encryption keys, and pass GDPR / HIPAA / SOC 2 audits with the same server you already own.
• No vendor lock-in -- Confluent's proprietary extensions (ksqlDB, Schema Registry behind auth, tiered storage on S3 with Confluent-specific configs) make migration painful. Vanilla Apache Kafka is portable across every cloud and bare-metal host.
• Connect to anything on your private network -- A self-hosted broker can live on the same private network as your databases, Redis, and application servers, with sub-millisecond latency and zero NAT traversal.
• Tunable for your workload -- Producer batching, log segment size, retention policies, compression codecs, and replication factor are all yours to tune. Managed services pick defaults that work for most users, but not for every user.

Cost Comparison: Self-Hosted vs. Managed Kafka

Prerequisites

Before you begin, make sure you have:

• A VPS running Ubuntu 24.04 LTS with root or sudo access
• SSH access to your server
• At least 8 GB of RAM (12 GB recommended for a single-node production broker; each node in a 3-node cluster should also have 8-12 GB)
• At least 50 GB of SSD/NVMe storage dedicated to Kafka logs (more for longer retention)
• A resolvable hostname or static IP for each broker (required for advertised.listeners)
• Ports 9092 (PLAINTEXT / SASL_SSL), 9093 (KRaft controller) reachable between cluster nodes

Recommended Plan: CloudCore Professional

Kafka brokers benefit from RAM (page cache for hot reads), fast disks (sequential writes for the log), and generous bandwidth. The CloudCore Professional plan is a strong fit for a single-node install or each node of a 3-node cluster:

- 6 vCPU cores

- 12 GB RAM

- 100 GB NVMe SSD

- Unmetered bandwidth

- EUR 19.99/month

For heavier workloads (100+ MB/s ingest, longer retention, or dozens of topics) step up to a plan with 200-400 GB storage, or add a block storage volume dedicated to log.dirs.

Connect to your server via SSH to get started:

ssh root@your-server-ip

Step 1: Update System Packages

Start by updating your package index and upgrading installed packages. This ensures you have the latest security patches before installing Java and Kafka.

sudo apt update && sudo apt upgrade -y

Expected output (abbreviated):

Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Reading package lists... Done
Building dependency tree... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Install a few utilities we will need along the way:

sudo apt install -y curl wget tar netcat-openbsd

If the kernel was updated, reboot:

sudo reboot

Step 2: Install JDK 17

Kafka 3.x runs on JDK 8, 11, or 17. JDK 17 is the current LTS and the recommended choice for new deployments. Ubuntu 24.04 ships OpenJDK 17 in its default repositories.

sudo apt install -y openjdk-17-jdk

Verify the installation:

java -version

Expected output:

openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-124.04)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-124.04, mixed mode, sharing)

Export JAVA_HOME so Kafka scripts can find the JDK:

echo "JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))" | sudo tee -a /etc/environment
source /etc/environment
echo $JAVA_HOME

Expected output:

/usr/lib/jvm/java-17-openjdk-amd64

Step 3: Create a Kafka System User

Running Kafka as root is a security risk. Create a dedicated service account that owns the binaries, configuration, and log directories.

sudo useradd --system --home /opt/kafka --shell /usr/sbin/nologin kafka

Create the data directory Kafka will use for its log segments:

sudo mkdir -p /var/lib/kafka/data
sudo chown -R kafka:kafka /var/lib/kafka

Step 4: Download and Install Kafka 3.7+

Download the latest 3.x release from the Apache mirrors. At the time of writing, Kafka 3.7.1 is the recommended stable release.

cd /tmp
wget https://downloads.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz

The filename encodes two versions: 2.13 is the Scala version Kafka was compiled against, and 3.7.1 is the Kafka version. Use the 2.13 Scala build unless you have a specific reason to use 2.12.

Extract and move the release into /opt/kafka:

tar -xzf kafka_2.13-3.7.1.tgz
sudo mv kafka_2.13-3.7.1 /opt/kafka
sudo chown -R kafka:kafka /opt/kafka

Verify the install:

/opt/kafka/bin/kafka-topics.sh --version

Expected output:

3.7.1

The /opt/kafka/bin directory contains all the CLI tools you will use day to day: kafka-topics.sh, kafka-console-producer.sh, kafka-console-consumer.sh, kafka-configs.sh, kafka-consumer-groups.sh, kafka-storage.sh, and kafka-server-start.sh.

Step 5: Format Storage with a Cluster ID (KRaft)

In KRaft mode, every cluster has a single UUID called the cluster ID that identifies it across all nodes. Before starting Kafka for the first time, you must generate this ID and use it to format the log directory. This writes a meta.properties file that the broker checks on every startup.

Generate a random cluster ID:

KAFKA_CLUSTER_ID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo $KAFKA_CLUSTER_ID

Expected output (your UUID will differ):

4L6g3nShT-eMCtK--X86sw

Save this value somewhere safe -- every node in the same cluster must be formatted with the exact same ID.

Kafka ships with a pre-made KRaft config template at /opt/kafka/config/kraft/server.properties. Format the storage directory using this template and your cluster ID:

sudo -u kafka /opt/kafka/bin/kafka-storage.sh format \
  -t $KAFKA_CLUSTER_ID \
  -c /opt/kafka/config/kraft/server.properties

Expected output:

Formatting /tmp/kraft-combined-logs with metadata.version 3.7-IV4.

We will override the log directory in the next step, but the formatter is happy to initialize the default path for now. If you see an error about the directory already existing, delete it and re-run:

sudo rm -rf /tmp/kraft-combined-logs

Step 6: Configure server.properties

The default KRaft config works for experimenting but needs tuning for production. Edit the file:

sudo nano /opt/kafka/config/kraft/server.properties

Set the following values. Replace 203.0.113.10 with your server's public IP (or private IP if the broker is only accessed internally).

# The role(s) this node plays. "broker,controller" runs both in a single process
(combined mode, recommended for small clusters). For large clusters, separate
controller and broker roles onto dedicated nodes.
process.roles=broker,controller
A unique numeric ID for this node within the cluster.
Must be different on every node of a multi-node cluster.
node.id=1
The controller quorum. For a single-node install this is just this node.
For a 3-node cluster you list all three controllers here (see Step 11).
controller.quorum.voters=1@localhost:9093
Listeners bound by this process.
PLAINTEXT on 9092 for clients, CONTROLLER on 9093 for KRaft metadata.
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
The host/port producers and consumers will connect to.
MUST be reachable from client machines. Use your public hostname or IP.
advertised.listeners=PLAINTEXT://203.0.113.10:9092
Map listener names to security protocols.
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
Name of the listener used for inter-broker communication.
inter.broker.listener.name=PLAINTEXT
Name of the listener used for KRaft controller communication.
controller.listener.names=CONTROLLER
Where Kafka writes its log segments. Point this at fast SSD/NVMe.
log.dirs=/var/lib/kafka/data
Default number of partitions for auto-created topics.
num.partitions=3
Replication settings for the internal __consumer_offsets topic.
Set to 3 on a 3-node cluster. Leave at 1 for single-node.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
Default retention: 7 days of data, then old segments are deleted.
log.retention.hours=168
Size-based retention cap per partition (in bytes). -1 disables it.
log.retention.bytes=-1
Roll a new log segment every 1 GB.
log.segment.bytes=1073741824
Check for segments eligible for deletion every 5 minutes.
log.retention.check.interval.ms=300000
Number of network/IO threads. Default 3/8 is usually fine; bump on busy brokers.
num.network.threads=3
num.io.threads=8
Socket buffer sizes.
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

Re-format the log directory now that log.dirs points at /var/lib/kafka/data:

sudo -u kafka /opt/kafka/bin/kafka-storage.sh format \
  -t $KAFKA_CLUSTER_ID \
  -c /opt/kafka/config/kraft/server.properties

Expected output:

Formatting /var/lib/kafka/data with metadata.version 3.7-IV4.

Tune the JVM heap. Kafka is happy with a modest heap because most of its memory goes to the OS page cache. On a 12 GB server, 4 GB heap leaves plenty of room for the kernel to cache hot log segments:

echo 'KAFKA_HEAP_OPTS="-Xms4G -Xmx4G"' | sudo tee /etc/default/kafka

Step 7: Create a systemd Service

Running Kafka under systemd gives you automatic restarts on crash, clean shutdown on reboot, and unified logging via journalctl.

Create the unit file:

sudo tee /etc/systemd/system/kafka.service > /dev/null <<'EOF'
[Unit]
Description=Apache Kafka (KRaft mode)
Documentation=https://kafka.apache.org/documentation/
Requires=network.target
After=network.target
[Service]
Type=simple
User=kafka
Group=kafka
EnvironmentFile=/etc/default/kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
Environment="LOG_DIR=/var/log/kafka"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=10
LimitNOFILE=100000
[Install]
WantedBy=multi-user.target
EOF

Create the log directory:

sudo mkdir -p /var/log/kafka
sudo chown -R kafka:kafka /var/log/kafka

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now kafka

Check status:

sudo systemctl status kafka

Expected output:

● kafka.service - Apache Kafka (KRaft mode)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-04-16 10:05:00 UTC; 10s ago
   Main PID: 1842 (java)
      Tasks: 78 (limit: 14236)
     Memory: 420.5M

Tail the logs to confirm startup:

sudo journalctl -u kafka -f

Look for a line similar to:

INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Press Ctrl+C to stop following.

Verify the broker is listening:

ss -tlnp | grep -E '9092|9093'

Expected output:

LISTEN 0 50 :9092 :* users:(("java",pid=1842,fd=...))
LISTEN 0 50 :9093 :* users:(("java",pid=1842,fd=...))

Step 8: Create Topics and Test Producer/Consumer

Kafka is running. Time to create a topic and push some events through it.

Create a topic named events with 3 partitions and replication factor 1 (we only have one broker):

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic events \
  --partitions 3 \
  --replication-factor 1

Expected output:

Created topic events.

List all topics to confirm:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Describe the topic to see partition layout:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe --topic events

Expected output:

Topic: events  TopicId: abc123  PartitionCount: 3  ReplicationFactor: 1
        Topic: events  Partition: 0  Leader: 1  Replicas: 1  Isr: 1
        Topic: events  Partition: 1  Leader: 1  Replicas: 1  Isr: 1
        Topic: events  Partition: 2  Leader: 1  Replicas: 1  Isr: 1

Produce Messages

Open a producer session and type a few messages, pressing Enter after each:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic events

>hello world
>kafka is running
>event three

Press Ctrl+C to exit the producer.

Consume Messages

In a second terminal (or the same one after exiting the producer), consume from the beginning of the topic:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic events \
  --from-beginning

Expected output:

hello world
kafka is running
event three

Press Ctrl+C to stop consuming. Note that --from-beginning reads the entire topic history -- without it, the consumer only sees new messages published after it started.

Consumer Groups

Kafka's killer feature is consumer groups: multiple consumers in the same group share the partitions of a topic, giving you horizontal scalability. Run a consumer with a group name:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic events \
  --group analytics-group

List consumer groups:

/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Inspect a group's current offsets and lag:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group analytics-group \
  --describe

Expected output:

GROUP           TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
analytics-group events  0          2               2               0
analytics-group events  1          1               1               0
analytics-group events  2          0               0               0

Step 9: Deploy a Web UI (Kafka UI / AKHQ)

Browsing topics, inspecting messages, and managing consumer groups from the CLI gets tedious fast. Two excellent open-source web UIs give you a point-and-click experience on top of any Kafka cluster:

• Kafka UI (also known as "provectus/kafka-ui" or its spiritual successor kafbat/kafka-ui) -- A lightweight Java/React UI. Fast, clean, and easy to deploy.
• AKHQ -- A more feature-rich Kafka manager with integrated support for Kafka Connect, Schema Registry, ksqlDB, and LDAP auth.

Install Docker if you do not already have it:

curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
newgrp docker

Run Kafka UI, pointing at your broker. Use host.docker.internal on Docker Desktop; on Linux, use the broker's public or private IP:

docker run -d \
  --name kafka-ui \
  --restart unless-stopped \
  -p 8080:8080 \
  -e KAFKA_CLUSTERS_0_NAME=local \
  -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=203.0.113.10:9092 \
  provectuslabs/kafka-ui:latest

Open http://your-server-ip:8080 in a browser. You will see the events topic, its partitions, consumer groups, and be able to produce/consume messages interactively.

For AKHQ instead, use this docker command:

docker run -d \
  --name akhq \
  --restart unless-stopped \
  -p 8081:8080 \
  -e AKHQ_CONFIGURATION='
akhq:
  connections:
    local:
      properties:
        bootstrap.servers: "203.0.113.10:9092"
' \
  tchiotludo/akhq:latest

Browse to http://your-server-ip:8081.

Put the UI behind a reverse proxy with authentication before exposing it publicly. An unauthenticated Kafka UI is equivalent to an unauthenticated Kafka broker: anyone can read and publish messages. Nginx with Basic Auth or OAuth2 Proxy works well.

Step 10: Enable SASL_SSL + SCRAM Authentication

The PLAINTEXT listener we started with is fine for a closed private network but unsafe on the public internet. Production deployments should use SASL_SSL: TLS for encryption in transit, and SASL/SCRAM-SHA-512 for username/password authentication.

Generate a Self-Signed CA and Broker Certificate

For a real deployment, obtain certificates from Let's Encrypt or an internal CA. For this guide we will generate a self-signed CA for demonstration:

sudo mkdir -p /opt/kafka/ssl
cd /opt/kafka/ssl
Create a CA
sudo openssl req -new -x509 -keyout ca-key.pem -out ca-cert.pem -days 3650 \
  -subj "/CN=Kafka-CA" -nodes
Create the broker keystore and key
sudo keytool -keystore kafka.server.keystore.jks -alias kafka \
  -validity 3650 -genkey -keyalg RSA \
  -dname "CN=kafka.example.com" \
  -storepass changeit -keypass changeit -noprompt
Create a certificate signing request
sudo keytool -keystore kafka.server.keystore.jks -alias kafka \
  -certreq -file ca-request.pem \
  -storepass changeit -noprompt
Sign it with the CA
sudo openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial \
  -in ca-request.pem -out ca-signed.pem -days 3650
Import the CA and signed cert back into the keystore
sudo keytool -keystore kafka.server.keystore.jks -alias CARoot \
  -import -file ca-cert.pem -storepass changeit -noprompt
sudo keytool -keystore kafka.server.keystore.jks -alias kafka \
  -import -file ca-signed.pem -storepass changeit -noprompt
Build the truststore
sudo keytool -keystore kafka.server.truststore.jks -alias CARoot \
  -import -file ca-cert.pem -storepass changeit -noprompt
sudo chown -R kafka:kafka /opt/kafka/ssl

Update server.properties

Edit /opt/kafka/config/kraft/server.properties and add/replace these sections:

# Add SASL_SSL listener on 9094 alongside existing PLAINTEXT on 9092
listeners=PLAINTEXT://:9092,CONTROLLER://:9093,SASL_SSL://:9094
advertised.listeners=PLAINTEXT://203.0.113.10:9092,SASL_SSL://kafka.example.com:9094
Enable SCRAM-SHA-512 on the SASL_SSL listener
listener.name.sasl_ssl.sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
SSL configuration for the SASL_SSL listener
ssl.keystore.location=/opt/kafka/ssl/kafka.server.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/opt/kafka/ssl/kafka.server.truststore.jks
ssl.truststore.password=changeit
ssl.client.auth=none

Restart Kafka:

sudo systemctl restart kafka

Create a SCRAM User

Create a user appuser with a password:

/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --add-config 'SCRAM-SHA-512=[password=S3cureP@ss!]' \
  --entity-type users --entity-name appuser

Expected output:

Completed updating config for user appuser.

Connect from a Client

Create client.properties on the client side:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="appuser" \
  password="S3cureP@ss!";
ssl.truststore.location=/path/to/ca-cert.pem

Produce to the SASL_SSL listener:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server kafka.example.com:9094 \
  --producer.config client.properties \
  --topic events

For fine-grained authorization, enable the built-in AclAuthorizer and use kafka-acls.sh to grant per-user, per-topic permissions. See the Kafka security documentation for the full set of SASL, SSL, and ACL options.

Step 11: Scale to a 3-Node Cluster

A single broker is fine for development, but production workloads need replication. Kafka's sweet spot is a 3-node cluster with replication factor 3: every partition has one leader and two followers, and the cluster survives the loss of any single node.

Provision Two More VPS

Repeat Steps 1-4 on two additional Ubuntu 24.04 servers. Assume they have IPs 203.0.113.11 and 203.0.113.12.

Configure Each Node

Edit /opt/kafka/config/kraft/server.properties on all three nodes. The key differences between nodes:

[email protected]:9093,[email protected]:9093,[email protected]:9093

And the replication factors should now all be 3:

offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
default.replication.factor=3
min.insync.replicas=2

Format Every Node with the Same Cluster ID

Generate the cluster ID once on node 1, then reuse it on nodes 2 and 3:

# On node 1 only
KAFKA_CLUSTER_ID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo $KAFKA_CLUSTER_ID

Copy that UUID to each node and run:

# On every node (same $KAFKA_CLUSTER_ID)
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format \
  -t <paste-the-same-uuid> \
  -c /opt/kafka/config/kraft/server.properties

Start All Three Brokers

sudo systemctl start kafka

Verify the cluster from any node:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 203.0.113.10:9092 describe --status

Expected output:

ClusterId:              4L6g3nShT-eMCtK--X86sw
LeaderId:               2
LeaderEpoch:            5
HighWatermark:          1234
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   12
CurrentVoters:          [1, 2, 3]
CurrentObservers:       []

Create a replicated topic:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 203.0.113.10:9092 \
  --create --topic orders \
  --partitions 6 --replication-factor 3

Describe it to confirm leader/follower distribution:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 203.0.113.10:9092 \
  --describe --topic orders

Every partition should show three replicas in the Isr (in-sync replicas) column. You now have a fault-tolerant cluster that can lose a node without losing data.

Kafka Connect: Extending the Platform

Kafka Connect is the official framework for streaming data between Kafka and other systems without writing custom code. Instead of a bespoke producer or consumer, you deploy pre-built connectors that handle the boilerplate of offset tracking, schema evolution, retries, and backoff.

Connect comes bundled with your Kafka install (/opt/kafka/bin/connect-distributed.sh). Popular connectors include:

• Debezium -- CDC from PostgreSQL, MySQL, MongoDB, Oracle, SQL Server into Kafka topics
• JDBC Source/Sink -- Generic database polling and writes
• Elasticsearch Sink -- Index Kafka messages into Elasticsearch/OpenSearch for search
• S3 Sink -- Archive topics to S3, MinIO, or other object storage in Parquet/Avro/JSON
• HTTP Sink -- Deliver messages to any webhook or REST endpoint
• ClickHouse Sink -- Stream analytics events into ClickHouse for sub-second queries

/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/connect-distributed.properties

Connector plugins are JAR files dropped into a plugin.path directory. Browse the full catalog at Confluent Hub. For most teams, Connect replaces large amounts of glue code with a few JSON config files POSTed to the worker's REST API.

Troubleshooting

Viewing Logs

Kafka's own application logs live at /var/log/kafka/. The systemd journal captures stdout/stderr:

sudo journalctl -u kafka -f
sudo journalctl -u kafka -n 200 --no-pager

FAQ

Do I need ZooKeeper to run Kafka?

No. Kafka 3.3+ supports KRaft mode, where cluster metadata is managed by a Raft quorum inside Kafka itself. Kafka 4.0 removes ZooKeeper support entirely. Every new deployment should use KRaft -- it has one less moving part, faster controller failover, and is the direction the project is heading. This guide uses KRaft exclusively.

What is the difference between a broker, a controller, and a node in KRaft?

A broker serves producer and consumer traffic: it owns partitions, handles reads and writes, and replicates data. A controller runs the Raft protocol that manages cluster metadata (topic list, partition assignments, in-sync replicas). In KRaft mode a single process can play both roles (process.roles=broker,controller, called "combined mode") which is ideal for clusters under ~10 nodes. Larger deployments separate roles: a few dedicated controller nodes (typically 3 or 5) and many broker-only nodes. The term node refers to any single Kafka process, regardless of role.

How much RAM and disk does Kafka need?

Kafka's throughput comes from the OS page cache, not the JVM heap, so more system RAM is almost always better. A broker with 12 GB RAM and a 4 GB Java heap leaves 8 GB for the kernel to cache recently-written log segments -- enough for most workloads. Disk is easier to reason about: retention_hours x peak_ingest_MBps x 3600 x replication_factor gives you the total bytes stored. For 10 MB/s ingest, 7-day retention, replication factor 3, you need roughly 18 TB per cluster (6 TB per node in a 3-node setup). Use fast SSD/NVMe: sequential write throughput is what bounds Kafka's ingest rate.

How does Kafka compare to RabbitMQ, NATS, and Redpanda?

RabbitMQ is a traditional AMQP message broker optimized for flexible routing (exchanges, queues, bindings) and per-message acknowledgments. It shines for RPC-style workloads, complex routing, and low-volume tasks with per-message delivery semantics. Kafka is a log; RabbitMQ is a queue.

NATS is a lightweight pub/sub system with optional JetStream persistence. NATS is much lower footprint and simpler to operate than Kafka, great for microservice request/reply and IoT telemetry, but lacks Kafka's ecosystem (Connect, Streams, ksqlDB) and mature retention/compaction story.

Redpanda is a Kafka-protocol-compatible broker written in C++ (no JVM) with a single-binary deployment model. It claims lower tail latency and simpler operations. If you want the Kafka API and ecosystem without JVM tuning, Redpanda is worth evaluating.

For most event-streaming, CDC, and analytics pipelines -- and anywhere you need the enormous connector ecosystem -- Apache Kafka remains the default choice.

Can I run Kafka and Kafka UI on the same VPS?

Yes. A single CloudCore Professional VPS can comfortably run a broker plus Kafka UI plus a small Connect worker. Just make sure Kafka's JVM heap plus Docker containers plus OS overhead stay under ~80% of system RAM to leave page cache room. For serious production workloads, give the broker its own server and run UIs on a separate lightweight VPS.

How do I back up Kafka data?

The simplest approach is MirrorMaker 2, Kafka's built-in cross-cluster replication tool. Run it against a second cluster in a different datacenter and every topic is mirrored continuously. For point-in-time snapshots, use the S3 Sink Connector from Kafka Connect to archive topics to object storage in Parquet or Avro -- those files can be restored into a new cluster if you ever need to. Avoid filesystem-level backups of log.dirs: you would need all brokers consistent at the same instant, and Kafka does not offer a "quiesce" command like a database.

Next Steps

With Kafka running on your VPS, here are useful directions to explore:

• Stream CDC from your database -- Deploy Debezium as a Kafka Connect plugin to tail your PostgreSQL or MySQL binlog into Kafka topics. Every row change becomes an event.

• Build a stream processing app -- Try Kafka Streams (Java/Kotlin library) or ksqlDB (SQL-on-Kafka) to compute rolling aggregations, joins, and windowed analytics without a separate cluster.

• Add Schema Registry -- Deploy the Apicurio Registry or Confluent's Schema Registry to enforce Avro/Protobuf/JSON Schema on your topics and make producer/consumer evolution safer.

• Monitor with Prometheus + Grafana -- Expose Kafka JMX metrics via the JMX Exporter, scrape them with Prometheus, and drop in a pre-built Grafana dashboard. Essential for spotting under-replicated partitions, broker saturation, and consumer lag before customers notice.

• Harden with ACLs -- Enable authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer and use kafka-acls.sh to grant per-user, per-topic READ/WRITE permissions. Combine with SASL_SSL from Step 10 for defence in depth.

• Compare to alternatives -- If you want a queue instead of a log, read How to Install RabbitMQ on Ubuntu. For a lightweight pub/sub, see How to Install NATS on Ubuntu. For a Kafka-compatible broker with a simpler operations story, try How to Install Redpanda on Ubuntu.

• Read the docs -- The Apache Kafka documentation is dense but excellent. Bookmark the "Configuration" and "Operations" sections.

Need a bigger broker?

Running Kafka at scale is one of the best reasons to step up from a shared-hosting plan to a dedicated VPS. Our CloudCore Professional plan gives you 6 vCPU, 12 GB RAM, 100 GB NVMe, and unmetered bandwidth for EUR 19.99/month -- the sweet spot for a production broker or each node of a 3-node cluster.

- NVMe storage for fast sequential log writes

- 12 GB RAM for generous page cache

- Unmetered bandwidth (no egress bill surprises)

- Ubuntu 24.04 LTS pre-installed

- Deploy three nodes in the same datacenter for low-latency replication

Launch your Kafka VPS now.