Spring Boot with Kafka

What You’ll Learn 📚

• Core concepts of Spring Boot and Kafka
• How to set up a simple Spring Boot application with Kafka
• Progressively complex examples to build your skills
• Common questions and troubleshooting tips

Introduction to Spring Boot and Kafka

Before we jump into the code, let’s understand what Spring Boot and Kafka are all about.

Spring Boot

Spring Boot is an open-source Java-based framework used to create stand-alone, production-grade Spring-based applications. It simplifies the process of setting up and developing new applications by providing defaults for code and annotation configuration.

Kafka

Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. It’s used for building real-time data pipelines and streaming apps. Kafka is designed to be durable, scalable, and fault-tolerant.

Key Terminology

• Producer: A client that sends messages to a Kafka topic.
• Consumer: A client that reads messages from a Kafka topic.
• Topic: A category or feed name to which records are published.
• Broker: A Kafka server that stores data and serves clients.

Setting Up Your Environment 🛠️

First things first, let’s set up our environment. You’ll need Java, Maven, and Kafka installed on your machine. Don’t worry, we’ll guide you through each step!

Step 1: Install Java

# Install Java (if not already installed)
sudo apt update
sudo apt install default-jdk

Step 2: Install Maven

# Install Maven
sudo apt install maven

Step 3: Install Kafka

# Download Kafka
wget https://downloads.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
# Extract the files
tar -xzf kafka_2.13-3.0.0.tgz
# Move to Kafka directory
cd kafka_2.13-3.0.0

💡 Tip: Make sure your Java and Maven installations are correctly set up by running java -version and mvn -version in your terminal.

Creating Your First Spring Boot Application with Kafka

Step 1: Create a Spring Boot Project

We’ll use Spring Initializr to create a new project. Go to start.spring.io and set the following:

• Project: Maven Project
• Language: Java
• Spring Boot: 2.5.4
• Dependencies: Spring Web, Spring for Apache Kafka

Download the project and unzip it.

Step 2: Configure Kafka

Open application.properties and add the following:

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.auto-offset-reset=earliest

🔍 Note: These properties configure your application to connect to a Kafka broker running locally.

Step 3: Create a Kafka Producer

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaProducer {

    private final KafkaTemplate kafkaTemplate;

    public KafkaProducer(KafkaTemplate kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendMessage(String message) {
        kafkaTemplate.send("my-topic", message);
    }
}

Step 4: Create a Kafka Consumer

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class KafkaConsumer {

    @KafkaListener(topics = "my-topic", groupId = "my-group")
    public void listen(String message) {
        System.out.println("Received Message: " + message);
    }
}

Running Your Application 🚀

To run your application, use the following command:

mvn spring-boot:run

Common Questions and Troubleshooting

1. Why isn’t my consumer receiving messages?

   Ensure your Kafka broker is running and the topic names match.

2. How do I create a new topic?

   bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

3. What if I get a ‘connection refused’ error?

   Check if your Kafka server is running and accessible at the specified port.

Troubleshooting Common Issues

⚠️ Warning: Ensure that your Kafka server is running before starting your Spring Boot application. Use bin/kafka-server-start.sh config/server.properties to start the Kafka server.

Practice Exercises

• Create a new topic and modify your producer and consumer to use this new topic.
• Experiment with different message formats, such as JSON.
• Try setting up multiple consumers for the same topic and observe the behavior.

Congratulations on completing this tutorial! 🎉 You’ve taken a big step in mastering Spring Boot with Kafka. Keep experimenting and building more complex applications. Happy coding! 💻