Apache Kafka has proven to be an extremely popular event streaming platform, as its scalable distributed architecture, high performance, and use cases, some key terms and concepts as below:

• Kafka clusters and Kafka brokers
• Kafka clients and servers
• Producers, and Consumers, and Consumer groups
• Kafka topics & Kafka partitions, offsets
• Kafka topic replication, leaders, and followers
• ZooKeeper or not

Typical Kafka Topology:

• Helm install bitnami/kafka

Here we use helm to install Kafka, then validate statefulset storage and cluster availability by creating a topic, producer, and consumer.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/kafka
kubectl create ns kafka
helm install zz-kafka . -n kafka

• Kafka-cluster status

kubectl get all | grep zz-kaf
pod/zz-kafka-client                                          1/1     Running   0               6m27s
pod/zz-kafka-controller-0                                    1/1     Running   0               8m52s
pod/zz-kafka-controller-1                                    1/1     Running   0               8m52s
pod/zz-kafka-controller-2                                    1/1     Running   0               8m52s
service/zz-kafka                                  ClusterIP   10.96.58.62             9092/TCP                        8m52s
service/zz-kafka-controller-headless              ClusterIP   None                    9094/TCP,9092/TCP,9093/TCP      8m52s
statefulset.apps/zz-kafka-controller                                    3/3     8m52s

kubectl get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
data-zz-kafka-controller-0   Bound    pvc-f3831a5c-c9cf-46bb-a47d-58ea80f82e28   8Gi        RWO            rook-ceph-block   9m19s
data-zz-kafka-controller-1   Bound    pvc-a45fd063-7979-4c0d-8ae4-93b4b4b0bf7f   8Gi        RWO            rook-ceph-block   9m19s
data-zz-kafka-controller-2   Bound    pvc-ed48b71d-81be-4067-b05f-9e8dc64fab04   8Gi        RWO            rook-ceph-block   9m19s

• Validate cluster by setting key

Create client.properties with SASL authentication details and copy to client.

kubectl run zz-kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.6.1-debian-11-r0 --namespace default --command -- sleep infinity
kubectl cp --namespace default client.properties zz-kafka-client:/tmp/client.properties

Open two bash windows to access the Kafka client.

kubectl exec --tty -i zz-kafka-client --namespace default -- bash

Window 1 (Producer):

kafka-console-producer.sh \
    --producer.config /tmp/client.properties \
    --broker-list zz-kafka-controller-0.zz-kafka-controller-headless.default.svc.cluster.local:9092,zz-kafka-controller-1.zz-kafka-controller-headless.default.svc.cluster.local:9092,zz-kafka-controller-2.zz-kafka-controller-headless.default.svc.cluster.local:9092 \
    --topic test

Window 2 (Consumer):

kafka-console-consumer.sh \
    --consumer.config /tmp/client.properties \
    --bootstrap-server zz-kafka.default.svc.cluster.local:9092 \
    --topic test \
    --from-beginning

Test topic and PRODUCER with CONSUMER.