"Welcome To Ashok IT" "Spring Boot and MicroServices" Topic : Apache Kafka Date : 15/02/2025 (Session - 125) _____________________________________________________________________________________________________________________________ Last Session ============ * We have been discussed about Distributed logging and tracing with help of sleuth and zipkin. * We added the dependencies into our Customer-Service, Address-Service. * We have two ids are available in distrubuted logging and tracing traceId >>>>>>>>>>>> unique indentifier for every request spanId >>>>>>>>>>>> Unique identiifier at each Microservice level * We ran the zipkin server the collect the trace information for every request. Today Session ============= * Kafka is just like messaging system Example : p1,p2(producers) --------------> KafkaSystem-------------------> c1,c2,c3(consumers) * It is distributed platform/application In production environment kafka reffered as kafka cluster A Cluster is madeup more than one kafka server. Each Kafka Server is referred as Broker. * Kafka is "Fault Tolerance" Ability of system to continue operating without interruption when one or more of its components fails. In Kafka Cluster messages are replicated with multiple brokers Replication Factor (When We are publishing the message and these messages are replicated into multiple clusters) * Kafka is "Scalabe System" We can add new brokers anytime We can increase number of consumers. * Kafka is an open-source, distributed streaming platform * It was originally developed by LinkedIn and later open-sourced as an Apache Software Foundation project. Kafka Components ================ 1.Producer ======== * Producers are responsible for publishing data to Kafka topics. * They send records (messages) to Kafka brokers. * Producers can be implemented in various programming languages. 2.Broker ======== * Brokers are the Kafka server instances within a Kafka cluster. * They receive records from producers, store these records in topics, and serve records to consumers. 3.Consumer ========== * Consumers are responsible for subscribing to Kafka topics and reading records from those topics. * They can be part of a consumer group. * Each consumer reads from one or more partitions. 4.Cluster ========= * Kafka clusters can have multiple brokers to achieve scalability and fault tolerance. 5.Topic ======= * Topics are logical channels or categories for data streams. * They are used to organize and categorize data. * Each topic can have one or more partitions. 6.Partitions ============ * Partitions are the basic unit of parallelism and distribution in Kafka. * Each topic is divided into partitions, allowing for parallel processing. * Partitions are stored on individual broker nodes. * Data within a partition is ordered and immutable. 7.offSet ======== * Each message in a partition is assigned a unique offset, which represents the message's position in the partition. * Consumers keep track of their current offset, which allows them to resume reading from where they left off. 8.Consumer groups ================= * A consumer group is a logical grouping of consumers that work together to consume records from one or more partitions of a topic. * Kafka ensures that each record is consumed by only one consumer within a group. 9.zookeeper =========== * In earlier versions of Kafka, ZooKeeper was used for managing cluster metadata and leader election. * In newer versions(since Kafka 2.8), ZooKeeper is not required, and Kafka uses its internal metadata management. 10.Producer API and Consumer API ================================ * Kafka provides producer and consumer APIs in various programming languages to interact with Kafka clusters. * These APIs are used to publish and consume records. Kafka Installation ================== Step-1 : Download the Apache Kafka Software with below URL https://kafka.apache.org/downloads Binary downloads:(kafka_2.12-3.6.0.tgz(asc,sha512)) Step-2 : After completing downloading we need to extract the zip folder into normal folder. The following are the below folders are available in extracted folder kafka_2.12-3.6.0 | |-------------------> bin | |----------------> *.bat | |-------------------> config | |----------------> *.properties | |-------------------> libs | |----------------> *.jar | |-------------------> logs | |----------------> *.logs | |-------------------> site-docs |----------------> documentation Step-3 : We need to install the offset explorer for viewing the Apache Kafka Components https://www.kafkatool.com/download.html Complete installation process just like normal software. Step-4 : Interacting with Apache Kafka CLI Interface using commands ****************************** Starting the Zookeeper Server ****************************** -> C:\kafka_2.12-3.6.0>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties >>>>>>> 2181 ******************* Apache Kafka Server ******************* -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-server-start.bat .\config\server.properties >>>>>>>>>>>>>>>9092 ****************************** Creating Topic in Apache Kafka ****************************** -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --create --topic sampletopic2 --partitions 3 --replication-factor 1 ******* OUTPUT ******* Created topic sampletopic2. ******************************* Listing the Apache Kafka Topics ******************************* -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --list ******* OUTPUT ******* sampletopic sampletopic2 sampletopics ******************************************* Viewing the Apache Kafka Partitions in Topic ******************************************* -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --describe --topic sampletopic ********************************** Creating the Apache Kafka Producer *********************************** -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic ashokit -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic sampletopic ********************************** Creating the Apache Kafka Consumer *********************************** -> C:\kafka_2.12-3.6.0>.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic ashokit --from-beginning