Installing an Apache Kafka Cluster on Debian 12

Quick Guide: Setting Up Apache Kafka on Debian 12

Apache Kafka is a powerful distributed event streaming platform designed for high-throughput and low-latency message distribution.

Setting up Kafka on Debian 12 requires careful configuration to enable reliable and scalable data streaming.

This guide provides a step-by-step walkthrough for installing and configuring an Apache Kafka cluster on Debian 12, highlighting essential steps and configurations.

Core Principles and Elements of Kafka's Architecture

Kafka’s architecture is based on several core components:

• Brokers: Kafka runs as a cluster comprised of one or more servers called brokers that manage message storage and distribution.
• Topics: Named categories to which records are sent and from which records are read.
• Producers: Applications that send records to Kafka topics.
• Consumers: Applications that read records from topics.
• Partitions: Kafka topics divided into sections for better distribution across multiple brokers.
• Replication: Each partition gets replicated, with one being responsible for read and write requests while others replicate data.
• Streams: A stream processing library for real-time streaming applications and microservices.
• ZooKeeper (optional): Previously used for metadata management, now replaced by Kafka Raft (KRaft) mode for better self-management.

Apache Kafka Installation on Debian 12

Install Java 8+ on Debian 12

Apache Kafka requires Java 8 or later to run. Follow these steps to install Java:

1. Update the package index:
   sudo apt update
2. Install the OpenJDK package:
   sudo apt install -y openjdk-11-jdk
3. Verify the Java installation:
   java -version

Download the Latest Kafka Release

1. Go to Apache Kafka’s download page to download the latest version.
2. Download Kafka using wget or curl:
   wget https://downloads.apache.org/kafka/3.x.x/kafka_2.13-3.x.x.tgz

Unpack the Kafka Binary Archive

Extract the downloaded archive:

tar -xzf kafka_2.13-3.x.x.tgz

Move the extracted folder to /opt:

sudo mv kafka_2.13-3.x.x /opt/kafka

Launching Kafka with Kafka Raft (KRaft)

KRaft mode eliminates the need for ZooKeeper, simplifying the Kafka cluster management. You can find the KRaft configuration files at /opt/kafka/config/kraft/.

Configure KRaft for a Kafka Cluster

1. Create a new configuration file or edit the existing server.properties:
   sudo nano /opt/kafka/config/kraft/server.properties
2. Add or modify the following properties:
   process.roles=broker,controller node.id=1 controller.quorum.voters=1@<broker-ip>:9093 listeners=PLAINTEXT://<broker-ip>:9092,CONTROLLER://<broker-ip>:9093 log.dirs=/var/lib/kafka/data

Modify Log Directory

Ensure Kafka logs are stored in a directory with sufficient disk space; by default, logs are stored in “/tmp/kraft-combined-logs” directory.

To ensure that logs are saved in a directory with enough space, update log.dirs in server.properties if needed:

log.dirs=/new/path/to/kafka-logs

Generate a Kafka Cluster ID

A cluster ID is what identifies a cluster in the KRaft mode.

1. Generate the cluster ID:
   /opt/kafka/bin/kafka-storage.sh random-uuid
2. Save the generated ID and format the log directory:
   /opt/kafka/bin/kafka-storage.sh format -t <cluster-id> -c /opt/kafka/config/kraft/server.properties

Convert Kafka Logs Directory to KRaft Format

Using the cluster ID we generated in the previous step, we can format the log directories. This is done for each Kafka broker to have a directory to save its log at (if there are multiple nodes in the cluster).

1. Using the cluster ID and the following command, format the log directory:
   /opt/kafka/bin/kafka-storage.sh format -t <uuid> -c /opt/kafka/config/kraft/server.properties
2. Replace the UUID for URaeRekUQAyy8wLMNX2Q-w for the command to look like this:
   /opt/kafka/bin/kafka-storage.sh format -t URaeRekUQAyy8wLMNX2Q-w \ -c /opt/kafka/config/kraft/server.properties

Alternatively, you can use the following command:

/opt/kafka/bin/kafka-storage.sh format \ -t /opt/kafka/bin/kafka-storage.sh random-uuid \ -c /opt/kafka/config/kraft/server.properties

Adjust Kafka Heap Size

For Kafka to have a stable performance, you can set the heap size accordingly; the heap size is basically the memory that belongs to the Java Virtual Machine (JVM) running Kafka. You need to ensure that the heap size can handle the amount of traffic your server would have to process.

One of the settings that use the memory and influences for how long does Kafka store the data is log.retention.hours, which is set by default for 7 days. Other parameters like log.retention.minutes, log.retention.ms, or log.retention.bytes can be used as well; these parameters can be updated in a /opt/kafka/config/kraft/server.properties file.

For example, to keep data for 8 hours, use the following line:

log.retention.hours=8

Then save and exit the file.

Starting the Kafka Broker

Start Kafka Broker in the Foreground

Run the Kafka broker manually:

/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties

Run Kafka Broker as a systemd Service

• Create a kafka.service file:

sudo nano /etc/systemd/system/kafka.service

• Add the following configuration:

[Unit] Description=Apache Kafka Service After=network.target

[Service] Type=simple ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties Restart=on-failure User=kafka

[Install] WantedBy=multi-user.target

• Reload systemd and start Kafka:

sudo systemctl daemon-reload sudo systemctl start kafka sudo systemctl enable kafka

Creating Topics, Sending, and Receiving Messages on Kafka

Your producers can publish messages to Kafka topics and consumers can subscribe to and read records from several topics under the right configuration.

Create a Topic Manually

To create a test topic on the Kafka server/broker use:

/opt/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server <broker-ip>:9092 --partitions 1 --replication-factor 1

View Available Kafka Topics

To view available Kafka topics, use:

/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server <broker-ip>:9092

Write (Produce) and Read (Consume) Messages on Kafka Topics

• To produce a message, use the following command:

/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \ --topic kafka-topic-test

When it runs, you will get such a prompt: >

• Type your message and press ENTER:

>Hello Kafka, this is my message

• To consume the message, use:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka-topic-test

Remove Kafka Topics

To delete one topic, use:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic <name-of-topic>

To delete all topics, use:

for i in /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092;do /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic $i; done

Setting Up a Three-Node Kafka Cluster

Configure multiple brokers by creating separate server.properties files for each node with unique node.id, listeners, and log. settings. Adjust controller.quorum.voters to include all nodes.

Kafka Cluster Management via UI

Tools like Kafka Manager, Conduktor, and Lenses.io provide graphical interfaces to monitor and manage Kafka clusters. Based on our experience, these tools provide easier administration.

Setting up Apache Kafka on Debian 12 with KRaft enables robust, decentralized message streaming and scalable data pipelines, essential for modern, data-driven applications.