How to Install Redpanda on Ubuntu 24.04 VPS: Kafka-Compatible Streaming Platform (No JVM, No ZooKeeper)

Redpanda is a streaming data platform that speaks the Kafka wire protocol but runs as a single C++ binary with no JVM, no ZooKeeper, and no external dependencies. It drops into any Kafka ecosystem as a direct replacement while delivering significantly lower latency, simpler operations, and a smaller hardware footprint. This guide walks you through installing Redpanda on an Ubuntu 24.04 VPS, from a single-node developer environment to a production-ready three-node cluster with Schema Registry, the Redpanda Console, TLS encryption, and SASL authentication.

Already running Kafka? Redpanda is a drop-in replacement. Your existing producers, consumers, Kafka Streams jobs, and Kafka Connect connectors work unchanged -- just point them at the Redpanda bootstrap servers.

Table of Contents

What is Redpanda?

Redpanda is an Apache Kafka-compatible streaming data platform written in C++ on top of the Seastar shard-per-core asynchronous framework. It implements the Kafka wire protocol, which means any Kafka client library, management tool, or stream processor can connect to Redpanda without code changes. What it replaces under the hood is the entire JVM-based Kafka broker stack: there is no kafka-server-start.sh, no ZooKeeper ensemble, no KRaft controller quorum to configure separately, and no garbage collector tuning. A Redpanda node is a single redpanda binary plus a systemd unit.

Redpanda uses a Raft-based replicated state machine per partition, so consensus and metadata management are built into the broker itself. Each broker pins a thread to each CPU core and uses a share-nothing model with direct I/O to NVMe storage, which is how it achieves single-digit-millisecond tail latencies even under load. It supports the full Kafka producer, consumer, admin, transactions, and idempotent-producer protocols, and it ships with a built-in HTTP Proxy (Pandaproxy) and a Confluent-compatible Schema Registry.

Common use cases map exactly to where you would otherwise deploy Kafka. Event-driven microservices publish domain events to topics that many consumers read independently. Change data capture (CDC) pipelines stream database mutations from Postgres or MySQL (via Debezium) into data warehouses. Real-time analytics stacks feed clickstream, IoT, and log data into stream processors like Apache Flink, ksqlDB, or Materialize. Observability pipelines aggregate application logs and metrics before fan-out to long-term storage. Machine learning feature pipelines publish feature updates for online model serving. Because Redpanda keeps full protocol compatibility, teams running Debezium, Kafka Streams, Kafka Connect, or the Confluent ecosystem can migrate with no consumer-side rewrite.

Why Self-Host Redpanda vs Redpanda Cloud or Confluent?

Redpanda Cloud and Confluent Cloud are both excellent managed offerings. They handle upgrades, replication, and scaling for you. They also bill per GB ingested, per GB stored, per partition, and sometimes per connected client. For many workloads, self-hosting Redpanda on your own VPS delivers the same protocol, the same tooling, and the same client libraries for a small fraction of the cost.

• Flat-rate cost -- A three-node Redpanda cluster on three CloudCore Professional plans costs under EUR 60/month with unmetered bandwidth. The same throughput on Confluent Cloud Standard easily runs into the hundreds of dollars per month once you factor in ingress, egress, and partition charges.
• No per-partition or per-connection fees -- Self-hosted Redpanda lets you create thousands of topics and tens of thousands of partitions without touching a billing meter. This matters for multi-tenant SaaS platforms where each tenant needs isolated topics.
• No cross-region egress surprises -- Managed offerings charge for data leaving their cloud. Run Redpanda on your VPS, and network egress is covered by the flat-rate plan. Producers and consumers in the same data center pay nothing.
• Data sovereignty and compliance -- Keep streams on infrastructure you control, in jurisdictions you choose. For GDPR workloads, healthcare events, or financial transaction logs, self-hosting simplifies compliance.
• Full configuration access -- Tune every broker property, plug in custom WASM transforms, enable tiered storage to any S3-compatible bucket, and run the exact Redpanda version you want. No waiting for a managed provider to upgrade.
• Co-located with your application -- Running brokers on the same VPC or private network as your producers and consumers eliminates network round trips. A local call stays under a millisecond; a round trip to Confluent Cloud easily adds 20-50 ms.
• No vendor lock-in -- The Redpanda binary is free, the protocol is Apache Kafka, and your topic data is just log segments on disk. You can migrate to self-managed Kafka, MSK, or Confluent at any time without changing application code.

Cost Comparison: Self-Hosted Redpanda vs Managed Offerings

Prerequisites

Before you begin, make sure you have:

• One or three VPS instances running Ubuntu 24.04 LTS with root or sudo access (one for the single-node walkthrough, three for the production cluster)
• SSH access to each server
• At least 2 vCPU and 4 GB of RAM per node for development; 4+ vCPU and 8+ GB of RAM per node for production
• NVMe or fast SSD storage with at least 50 GB free per node (Redpanda is I/O sensitive -- avoid spinning disks)
• A private network or firewall rules that allow the cluster nodes to reach each other on TCP ports 9092 (Kafka API), 33145 (RPC), 9644 (Admin API), 8081 (Schema Registry), and 8082 (HTTP Proxy)

Recommended Plan: CloudCore Professional

For a production-ready three-node Redpanda cluster, we recommend three CloudCore Professional instances:

- 6 vCPU cores

- 12 GB RAM

- 100 GB NVMe SSD

- Unmetered bandwidth

- EUR 19.99/month per node (EUR 59.97/month for the cluster)

This provides enough CPU to dedicate cores to Redpanda's reactor, plus headroom for Redpanda Console and Schema Registry on the same nodes. For heavier throughput (500 MB/s+), step up to 8 vCPU / 32 GB RAM plans.

Connect to your server via SSH to get started:

ssh root@your-server-ip

Step 1: Update System Packages and Tune the Host

Start by updating your package index and upgrading installed packages. Redpanda makes aggressive use of modern kernel features, so having the latest patches matters.

sudo apt update && sudo apt upgrade -y

Install a few utilities used later in the guide:

sudo apt install -y curl gnupg ca-certificates lsb-release ufw

If the kernel was upgraded, reboot before continuing:

sudo reboot

Redpanda expects a few kernel and filesystem characteristics. The rpk CLI provides a redpanda tune command that applies all of them automatically in Step 4, but you can verify your disk layout now: Redpanda's data directory should live on an XFS or ext4 filesystem on NVMe or fast SSD. Avoid network-attached storage, and avoid filesystems shared with other high-I/O workloads.

Step 2: Add the Redpanda APT Repository

Redpanda ships signed APT packages from packages.redpanda.com. The easiest way to add the repository is via the upstream install script, which handles both the GPG key and the apt source file.

curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash

Expected output (abbreviated):

Executing the setup script for the 'redpanda/redpanda' repository ...
Importing 'redpanda-redpanda-archive-keyring.gpg' into '/usr/share/keyrings' ...
Installing the debian-archive-keyring package...
Creating apt config file list: redpanda-redpanda.list ...
Running apt-get update ... done.
The repository is set up! You can now install packages.

The script installs the archive keyring at /usr/share/keyrings/redpanda-redpanda-archive-keyring.gpg and creates /etc/apt/sources.list.d/redpanda-redpanda.list, which points at the stable channel for your Ubuntu release. If you prefer to add the repository manually, download the keyring and write the apt source file yourself -- the upstream docs at docs.redpanda.com include the manual commands.

Step 3: Install Redpanda and the rpk CLI

Install the redpanda package. This pulls in both the broker binary and rpk, the Redpanda command-line tool used for cluster administration, topic management, and diagnostics.

sudo apt install -y redpanda

Expected output:

The following NEW packages will be installed:
  redpanda
...
Setting up redpanda (24.3.x-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/redpanda.service ...

Verify the installed versions:

redpanda --version
rpk version

Expected output:

v24.3.3 (rev abc1234)
v24.3.3

The install places the broker binary at /opt/redpanda/bin/redpanda, installs rpk at /usr/bin/rpk, creates a dedicated redpanda system user, and sets up a systemd unit at /lib/systemd/system/redpanda.service. The service is not started automatically -- you bootstrap it after generating a configuration in the next step.

Step 4: Single-Node Developer Install

For local development, experimentation, or CI pipelines, a single-node Redpanda setup is the fastest way to start producing and consuming messages. Use rpk redpanda mode dev to generate a minimal configuration optimized for a single machine.

sudo rpk redpanda mode dev

Expected output:

Writing 'dev' mode defaults to /etc/redpanda/redpanda.yaml

This writes a /etc/redpanda/redpanda.yaml with developer defaults: a single seed server pointing at itself, the admin API on 127.0.0.1:9644, and the Kafka API on 0.0.0.0:9092. It also disables some safety checks (like the minimum RAM-per-core guard) so you can run on smaller VPS plans.

Apply the recommended host tuning. In dev mode most tuners are noops, but running the command is a good habit:

sudo rpk redpanda tune all

Now start the service:

sudo systemctl start redpanda
sudo systemctl enable redpanda

Verify that the broker is healthy:

sudo systemctl status redpanda

Expected output:

● redpanda.service - Redpanda, the fastest queue in the West.
     Loaded: loaded (/lib/systemd/system/redpanda.service; enabled; preset: enabled)
     Active: active (running) since ...
   Main PID: 1234 (redpanda)
      Tasks: 12 (limit: 14236)
     Memory: 1.2G
     CGroup: /system.slice/redpanda.service
             └─1234 /opt/redpanda/bin/redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml

Query cluster status via rpk:

rpk cluster info

Expected output:

CLUSTER
=======
redpanda.your-hostname
BROKERS
=======
ID    HOST          PORT
0*    127.0.0.1     9092

The asterisk next to the broker ID marks the current Raft leader of the controller partition. You now have a working Kafka-compatible broker. Skip ahead to Step 6 to create topics, or continue with Step 5 to deploy a production three-node cluster.

Step 5: Deploy a Three-Node Production Cluster

Production deployments should run at least three brokers so that topic partitions can be replicated with a replication factor of 3 and survive the loss of a single node without downtime or data loss. Repeat Steps 1-3 on each of your three VPS instances before continuing.

Assume the following hostnames and private IPs for the walkthrough:

• redpanda-1 -- 10.0.0.11
• redpanda-2 -- 10.0.0.12
• redpanda-3 -- 10.0.0.13

Bootstrap the first node

On redpanda-1, switch from dev mode to production mode:

sudo rpk redpanda mode production

Then initialize the node's configuration with its identity and the list of seed servers:

sudo rpk redpanda config bootstrap \
  --id 0 \
  --self 10.0.0.11 \
  --ips 10.0.0.11,10.0.0.12,10.0.0.13

This command writes /etc/redpanda/redpanda.yaml with a unique node ID, binds the Kafka, admin, and RPC listeners to the node's private IP, and populates the seed_servers section with all three nodes. Redpanda uses the seed-server list to form the controller Raft group on first boot and to discover peers on restart.

Apply production host tuning. This enables the CPU governor, disables transparent huge pages, tunes NIC interrupts, and applies I/O scheduler changes:

sudo rpk redpanda tune all

Start the broker:

sudo systemctl start redpanda
sudo systemctl enable redpanda

Bootstrap the remaining nodes

On redpanda-2:

sudo rpk redpanda mode production
sudo rpk redpanda config bootstrap \
  --id 1 \
  --self 10.0.0.12 \
  --ips 10.0.0.11,10.0.0.12,10.0.0.13
sudo rpk redpanda tune all
sudo systemctl start redpanda
sudo systemctl enable redpanda

On redpanda-3:

sudo rpk redpanda mode production
sudo rpk redpanda config bootstrap \
  --id 2 \
  --self 10.0.0.13 \
  --ips 10.0.0.11,10.0.0.12,10.0.0.13
sudo rpk redpanda tune all
sudo systemctl start redpanda
sudo systemctl enable redpanda

Verify the cluster

From any node, run:

rpk cluster info --brokers 10.0.0.11:9092,10.0.0.12:9092,10.0.0.13:9092

Expected output:

CLUSTER
=======
redpanda.production
BROKERS
=======
ID    HOST         PORT
0*    10.0.0.11    9092
1     10.0.0.12    9092
2     10.0.0.13    9092

All three brokers appear and have joined the controller Raft group. You can also check cluster health at any time:

rpk cluster health

Expected output:

CLUSTER HEALTH OVERVIEW
=======================
Healthy:                     true
Unhealthy reasons:           []
Controller ID:               0
All nodes:                   [0 1 2]
Nodes down:                  []
Leaderless partitions:       []
Under-replicated partitions: []

To avoid repeating --brokers on every command, save a profile:

rpk profile create prod \
  --from-profile <(echo '{"kafka_api":{"brokers":["10.0.0.11:9092","10.0.0.12:9092","10.0.0.13:9092"]}}')
rpk profile use prod

Step 6: Create Topics and Test Producers and Consumers

With the cluster running, create a topic. For a three-node production cluster, use a replication factor of 3 and at least as many partitions as you expect concurrent consumers.

rpk topic create orders --partitions 6 --replicas 3

Expected output:

TOPIC   STATUS
orders  OK

List topics to confirm:

rpk topic list

Expected output:

NAME    PARTITIONS  REPLICAS
orders  6           3

Describe the topic to see per-partition leader and replica assignments:

rpk topic describe orders

Expected output (abbreviated):

SUMMARY
=======
NAME        orders
PARTITIONS  6
REPLICAS    3
PARTITIONS
==========
PARTITION  LEADER  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          0       [0 1 2]   0                 0
1          1       [0 1 2]   0                 0
2          2       [0 1 2]   0                 0
3          0       [0 1 2]   0                 0
4          1       [0 1 2]   0                 0
5          2       [0 1 2]   0                 0

Produce a message

The rpk topic produce command reads lines from stdin and publishes each as a record:

echo '{"order_id": 1001, "amount": 49.99}' | rpk topic produce orders

Expected output:

Produced to partition 3 at offset 0 with timestamp 1713268800000.

Consume messages

In a separate terminal, start a consumer:

rpk topic consume orders --num 1

Expected output:

{
  "topic": "orders",
  "key": null,
  "value": "{\"order_id\": 1001, \"amount\": 49.99}",
  "timestamp": 1713268800000,
  "partition": 3,
  "offset": 0
}

Use --offset start to replay from the beginning of the topic, or -g my-group to consume as part of a named consumer group with committed offsets:

rpk topic consume orders -g analytics --offset start

Because Redpanda implements the Kafka protocol, any standard Kafka client library works unchanged. The same Java, Python (confluent-kafka, kafka-python, aiokafka), Go (franz-go, sarama), Node.js (kafkajs), and Rust (rdkafka) clients produce and consume against your cluster by pointing bootstrap.servers at 10.0.0.11:9092,10.0.0.12:9092,10.0.0.13:9092.

Step 7: Enable the Schema Registry

The Schema Registry provides a REST API for registering and retrieving Avro, Protobuf, and JSON Schema definitions. Clients fetch schemas by ID when serializing and deserializing records, which keeps producer and consumer contracts compatible as your data model evolves. Redpanda bundles a Confluent-compatible Schema Registry in every broker -- no separate service to install.

Edit /etc/redpanda/redpanda.yaml on each node and ensure the schema_registry section is present:

schema_registry:
  schema_registry_api:
    - address: 0.0.0.0
      port: 8081

Restart Redpanda on each node:

sudo systemctl restart redpanda

Test the registry:

curl http://10.0.0.11:8081/subjects

Expected output (empty cluster):

[]

Register an Avro schema for the orders topic:

curl -X POST http://10.0.0.11:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schemaType": "AVRO",
    "schema": "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"order_id\",\"type\":\"long\"},{\"name\":\"amount\",\"type\":\"double\"}]}"
  }'

Expected output:

{"id": 1}

Retrieve the registered schema:

curl http://10.0.0.11:8081/subjects/orders-value/versions/1

Producers now serialize records using schema ID 1, and consumers fetch the schema from any broker before deserializing. The registry stores its data in an internal _schemas topic that is itself replicated across the cluster.

Step 8: Deploy Redpanda Console

Redpanda Console is a web UI for browsing topics, inspecting messages, managing consumer groups, and viewing Schema Registry contents. It runs as a separate process and connects to your cluster as a regular Kafka client.

Install Console on one of your nodes (or a separate admin server):

sudo apt install -y redpanda-console

Create a configuration file at /etc/redpanda-console/redpanda-console.yaml:

kafka:
  brokers:
    - 10.0.0.11:9092
    - 10.0.0.12:9092
    - 10.0.0.13:9092
  schemaRegistry:
    enabled: true
    urls:
      - http://10.0.0.11:8081
redpanda:
  adminApi:
    enabled: true
    urls:
      - http://10.0.0.11:9644
      - http://10.0.0.12:9644
      - http://10.0.0.13:9644
server:
  listenPort: 8080

Enable and start the service:

sudo systemctl enable --now redpanda-console

Verify:

sudo systemctl status redpanda-console
curl http://127.0.0.1:8080

Open http://your-server-ip:8080 in a browser (or front it with an Nginx reverse proxy for TLS -- see Step 9) to access the UI. You can browse the orders topic, peek at individual messages, inspect consumer group lag, view broker metrics, and manage ACLs from the Console.

Step 9: Secure the Cluster with TLS and SASL

For any cluster exposed beyond a private network, enable TLS encryption on the Kafka API and require SASL authentication for clients. Redpanda supports SASL/SCRAM (recommended for self-hosted deployments) and SASL/GSSAPI (Kerberos).

Generate TLS certificates

For production, use certificates signed by a trusted internal CA or Let's Encrypt. For this walkthrough, generate a self-signed CA and per-broker certificates with openssl:

mkdir -p /etc/redpanda/certs
cd /etc/redpanda/certs
CA
openssl req -new -x509 -days 3650 -nodes -out ca.crt -keyout ca.key \
  -subj "/CN=Redpanda Internal CA"
Broker cert (repeat with each node's hostname/IP)
openssl req -new -nodes -out broker.csr -keyout broker.key \
  -subj "/CN=redpanda-1"
openssl x509 -req -in broker.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out broker.crt -days 365 \
  -extfile <(printf "subjectAltName=DNS:redpanda-1,IP:10.0.0.11")

Distribute the CA certificate and each node's matching broker certificate and key to /etc/redpanda/certs/ on every broker. Set ownership to the redpanda user:

sudo chown -R redpanda:redpanda /etc/redpanda/certs
sudo chmod 600 /etc/redpanda/certs/*.key

Enable TLS on the Kafka API

Edit /etc/redpanda/redpanda.yaml on each node:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: external
  kafka_api_tls:
    - name: external
      enabled: true
      cert_file: /etc/redpanda/certs/broker.crt
      key_file: /etc/redpanda/certs/broker.key
      truststore_file: /etc/redpanda/certs/ca.crt
      require_client_auth: false
  enable_sasl: true

Restart each broker:

sudo systemctl restart redpanda

Create a SASL user

Create a superuser for bootstrapping, then application-specific users:

rpk cluster config set superusers "[admin]"
rpk acl user create admin -p 'ChangeMeStrongPassword!' \
  --mechanism SCRAM-SHA-256
rpk acl user create app-producer -p 'AppProducerSecret!' \
  --mechanism SCRAM-SHA-256
rpk acl user create app-consumer -p 'AppConsumerSecret!' \
  --mechanism SCRAM-SHA-256

Grant least-privilege ACLs:

rpk acl create --allow-principal User:app-producer \
  --operation write --topic orders
rpk acl create --allow-principal User:app-consumer \
  --operation read --topic orders \
  --operation describe --topic orders \
  --operation read --group analytics

Connect with TLS and SASL from the CLI

Save the credentials in an rpk profile:

rpk profile create secure \
  --set 'kafka_api.brokers=["10.0.0.11:9092","10.0.0.12:9092","10.0.0.13:9092"]' \
  --set kafka_api.tls.ca_file=/etc/redpanda/certs/ca.crt \
  --set kafka_api.sasl.user=admin \
  --set kafka_api.sasl.password='ChangeMeStrongPassword!' \
  --set kafka_api.sasl.mechanism=SCRAM-SHA-256
rpk profile use secure
rpk cluster info

All subsequent rpk commands will authenticate and encrypt automatically. Update your application clients to use the same TLS CA file, SASL mechanism, and credentials.

Step 10: Connect with Kafka Connect

Kafka Connect is the ecosystem's standard framework for streaming data into and out of external systems. Debezium (Postgres, MySQL, MongoDB CDC), JDBC sink and source, S3 sink, Elasticsearch sink, HTTP sink, and hundreds of other connectors run unchanged against Redpanda because they speak the Kafka protocol.

Install a distributed Connect worker on a separate Ubuntu VPS (do not run Connect on your brokers -- it is a JVM process and competes for CPU with Redpanda's reactor).

Install Java and download Kafka Connect:

sudo apt install -y openjdk-17-jre-headless
cd /opt
sudo curl -LO https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
sudo tar xzf kafka_2.13-3.7.0.tgz
sudo mv kafka_2.13-3.7.0 kafka

Create a worker config at /opt/kafka/config/connect-distributed.properties:

bootstrap.servers=10.0.0.11:9092,10.0.0.12:9092,10.0.0.13:9092
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://10.0.0.11:8081
offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
config.storage.topic=connect-configs
config.storage.replication.factor=3
status.storage.topic=connect-status
status.storage.replication.factor=3
plugin.path=/opt/kafka/connectors
SASL/TLS (if enabled)
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" password="ChangeMeStrongPassword!";
ssl.truststore.location=/etc/redpanda/certs/ca.p12
ssl.truststore.type=PKCS12

Create the internal topics that Connect needs:

rpk topic create connect-offsets --partitions 25 --replicas 3 \
  --config cleanup.policy=compact
rpk topic create connect-configs --partitions 1 --replicas 3 \
  --config cleanup.policy=compact
rpk topic create connect-status --partitions 5 --replicas 3 \
  --config cleanup.policy=compact

Download a connector (Debezium Postgres as an example):

sudo mkdir -p /opt/kafka/connectors
cd /opt/kafka/connectors
sudo curl -LO https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.7.3.Final/debezium-connector-postgres-2.7.3.Final-plugin.tar.gz
sudo tar xzf debezium-connector-postgres-2.7.3.Final-plugin.tar.gz

Start the worker:

/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/connect-distributed.properties

Register a connector via the REST API:

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
  "name": "postgres-orders",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres.internal",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "secret",
    "database.dbname": "shop",
    "topic.prefix": "pg",
    "table.include.list": "public.orders"
  }
}'

Debezium will now stream Postgres CDC events into the pg.public.orders topic on Redpanda, where any consumer (or another Connect sink into S3, Snowflake, Elasticsearch, etc.) can read them.

Performance Tuning

Out of the box, a three-node cluster on CloudCore Professional handles tens of thousands of messages per second with single-digit-millisecond producer acknowledgement latency. A few knobs matter for pushing further:

Reserve cores for the reactor

Redpanda pins one thread per CPU core. On a 6 vCPU node, the default is to use all 6. If you run Console, Schema Registry on the same node, or other services, reserve cores for the OS:

redpanda:
  reserve_memory: 1GB
rpk:
  additional_start_flags:
    - "--smp=4"        # Use 4 cores for Redpanda, leave 2 for OS/Console
    - "--memory=8G"    # Cap Redpanda memory

Tune flush behavior

Redpanda fsyncs on every produce by default in production mode. For lower durability requirements (e.g., analytics ingest where a few dropped messages on a crash are acceptable), set:

rpk cluster config set log_segment_ms 60000
rpk cluster config set raft_replica_max_pending_flush_bytes 1048576

Partition sizing

Rule of thumb: aim for 6-10 partitions per CPU core across the cluster. A 3-node cluster with 6 vCPU per node (18 cores total) comfortably handles 100-200 partitions. Over-partitioning adds metadata overhead; under-partitioning caps consumer parallelism.

Enable tiered storage (optional)

For long retention without growing local disk, enable tiered storage to any S3-compatible bucket:

rpk cluster config set cloud_storage_enabled true
rpk cluster config set cloud_storage_access_key "AKIA..."
rpk cluster config set cloud_storage_secret_key "..."
rpk cluster config set cloud_storage_region "us-east-1"
rpk cluster config set cloud_storage_bucket "my-redpanda-archive"

Local disk now holds a hot window (controlled by retention.local.target.bytes per topic) while older segments are offloaded to S3 and re-fetched on demand for consumers doing historical reads.

Troubleshooting

Viewing logs

The broker log is the most useful diagnostic:

sudo journalctl -u redpanda -f
sudo journalctl -u redpanda -n 200 --no-pager

For Console:

sudo journalctl -u redpanda-console -f

FAQ

Is Redpanda actually Kafka-compatible?

Yes. Redpanda implements the Apache Kafka wire protocol up to and including the latest KIPs for transactions, idempotent producers, and consumer groups. All of the standard client libraries -- Java, librdkafka (Python, Go, Node.js, Rust bindings), Sarama, franz-go, kafkajs -- work unchanged. Kafka Streams and Kafka Connect both run against Redpanda in production. The only thing you will not find is ZooKeeper or the KRaft controller process, because Redpanda embeds Raft-based consensus directly in the broker.

Do I need ZooKeeper or KRaft?

No. Redpanda uses its own Raft implementation internally for metadata (the controller log) and for each partition. There is no external consensus system to deploy, monitor, or upgrade. This is one of the largest operational wins over running Kafka yourself: your cluster is just N broker processes.

Can I migrate from Kafka to Redpanda without downtime?

Most teams use MirrorMaker 2 (the same tool used for Kafka-to-Kafka replication) to mirror topics from an existing Kafka cluster into a new Redpanda cluster, cut consumers over, and then cut producers over. Because the protocols match, the consumer-side cutover is usually a bootstrap.servers config change. Follow the detailed migration playbook at docs.redpanda.com.

How does Redpanda compare to NATS, RabbitMQ, and Kafka?

Redpanda is a drop-in replacement for Kafka and fits the same use cases: high-throughput event streaming, replayable logs, change data capture, and stream processing with exactly-once semantics. It is not an in-memory pub/sub like NATS, and it is not a traditional work-queue broker like RabbitMQ. If your application already uses or would benefit from Kafka semantics -- durable, ordered, replayable, partitioned logs with consumer groups -- Redpanda gives you that with a simpler operational footprint. For lighter-weight messaging needs, see our guides on installing NATS on Ubuntu and installing RabbitMQ on Ubuntu. For the vanilla Apache Kafka deployment, see installing Kafka on Ubuntu.

What throughput can I expect on a three-node CloudCore Professional cluster?

With default configuration and a reasonable partition count (32-64 partitions on the hot topics), expect roughly 200-400 MB/s of aggregate produce throughput at single-digit-millisecond p99 latency. Tuning producer batch sizes, linger, and compression (use zstd) can push this higher. Tiered storage lets you keep retention weeks long without scaling local disk.

How do I monitor Redpanda in production?

Redpanda exposes Prometheus-format metrics on the admin API at /metrics. Scrape http://redpanda-node:9644/metrics from Prometheus and import the official Redpanda Grafana dashboards. Key metrics to alert on: redpanda_kafka_under_replicated_replicas (should be 0), redpanda_storage_disk_free_bytes (disk headroom), redpanda_kafka_request_latency_seconds (p99 produce/fetch), and redpanda_cpu_busy_seconds_total (per-core utilization).

Next Steps

Now that Redpanda is running on your VPS, here are recommended next steps:

• Build a stream processor with Benthos -- Point Benthos at your Redpanda cluster for declarative stream processing pipelines in YAML, no code required.
• Wire up Debezium CDC -- Stream changes from Postgres or MySQL into Redpanda topics and fan them out to data warehouses, caches, and search indexes.
• Add tiered storage -- Enable S3-backed tiered storage so topics can retain months of history without scaling local disk. See the Redpanda tiered storage docs.
• Deploy Materialize or Apache Flink -- Run real-time SQL analytics over your Redpanda topics for dashboards and alerts.
• Set up Prometheus and Grafana -- Scrape the admin API /metrics endpoint and import the official Redpanda dashboards for broker health, partition status, and throughput visibility.
• Compare alternatives -- Review how to install Kafka on Ubuntu, NATS on Ubuntu, and RabbitMQ on Ubuntu to choose the right messaging platform for your workload.

Run Redpanda on CloudCore Professional

Redpanda shines on fast NVMe storage with dedicated CPU cores. Our CloudCore Professional plans deliver exactly that, with unmetered bandwidth and a private network for broker-to-broker traffic.

- 6 vCPU cores per node

- 12 GB RAM per node

- 100 GB NVMe SSD per node

- Unmetered bandwidth on a private network

- From EUR 19.99/month per node

Launch Your Redpanda Cluster and start streaming in under 30 minutes.