Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 108 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,118 @@
# spring-boot-kafka
# Spring Boot with Apache Kafka

### Install and Run Kafka:
To download Kafka, go to the Kafka website https://kafka.apache.org/downloads. Extract the contents of this compressed file into a folder of your preference.
Inside the Kafka directory, go to the bin folder. Here you’ll find many bash scripts that will be useful for running a Kafka application. If you are using Windows, you also have the same scripts inside the windows folder. This tutorial uses Linux commands, but you just need to use the equivalent Windows version if you’re running a Microsoft OS.
A hands-on Spring Boot demo that produces and consumes messages with Apache Kafka using the publish/subscribe model.

### Start Zookeeper to Manage Your Kafka Cluster:
Let’s start a Zookeeper instance! Inside the bin folder in your Kafka directory, run the following command:
This tutorial covers the basics of Kafka and walks through a small Java app with a REST producer, a Kafka consumer, and an endpoint to inspect consumed messages.

`./zookeeper-server-start.sh ../config/zookeeper.properties`
## Features

### Run a Kafka Broker:
The next step is to run the broker itself. From another terminal, run the following command from the bin folder:
- Produce messages to a Kafka topic over HTTP
- Consume messages from the same topic with `@KafkaListener`
- List messages the consumer has received so far
- JSON serialization for producer payloads (`User` objects)

`./kafka-server-start.sh ../config/server.properties`
## Tech stack

### Create a Kafka Topic:
Now that you have the broker and Zookeeper running, you can specify a topic to start sending messages from a producer. You’re going to run a command inside the bin folder, just like you did in the previous steps:
| Component | Version / detail |
|------------------|-------------------------|
| Java | 8 |
| Spring Boot | 2.2.4.RELEASE |
| Spring Kafka | (managed by Boot) |
| Kafka broker | `localhost:9092` |
| Topic | `myTopic` |
| Consumer group | `kafka-sandbox` |
| App port | `8081` |

`./kafka-topics.sh --create --topic myTopic -zookeeper \ localhost:2181 --replication-factor 1 --partitions 1`
## Prerequisites

### import Spring-boot-kafka project:
`$ mvn clean install`
`$ mvn spring-boot:run`
Or spring-boot-kafka project import in IDE
- JDK 8+
- Maven 3.x
- Apache Kafka (classic Zookeeper mode — download from [kafka.apache.org/downloads](https://kafka.apache.org/downloads))

### Run spring-boot-kafka project
Your Java app now has both a Kafka producer and a consumer, so let’s test it all together! Restart your application, and go to postman or browser and enter URL for cunsumer messages by kafka topic `http://localhost:8080/kafka/messages`
## Project structure

Right now, no information is being returned. The reason is pretty simple: your consumer is configured only to receive new messages and you haven’t sent a new message yet. Let’s fix this problem by going to your web browser and accessing `http://localhost:8080/kafka/produce?message=Message sent by my App!.` and send message by producer to kafka broker.
```
spring-boot-with-apache-kafka/
└── spring-boot-kafka/
├── pom.xml
└── src/main/java/com/chat/kafka/
├── SpringBootKafkaApplication.java
├── controller/KafkaController.java
├── producer/ProducerConfiguration.java
├── consume/ConsumerConfiguration.java
├── consume/MyTopicConsumer.java
└── model/User.java
```

## 1. Install and start Kafka

Extract Kafka to a directory of your choice. Commands below assume you are inside the Kafka `bin` folder (use the scripts under `bin/windows` on Windows).

### Start Zookeeper

```bash
./zookeeper-server-start.sh ../config/zookeeper.properties
```

### Start the Kafka broker

In a second terminal:

```bash
./kafka-server-start.sh ../config/server.properties
```

### Create the topic

In a third terminal:

```bash
./kafka-topics.sh --create --topic myTopic \
--zookeeper localhost:2181 \
--replication-factor 1 \
--partitions 1
```

> **Note:** Newer Kafka versions prefer `--bootstrap-server localhost:9092` instead of `--zookeeper`. Use the flag that matches your Kafka install.

## 2. Run the Spring Boot app

```bash
cd spring-boot-kafka
mvn clean install
mvn spring-boot:run
```

Or import `spring-boot-kafka` into your IDE and run `SpringBootKafkaApplication`.

The app listens on **http://localhost:8081**.

## 3. Try the APIs

| Action | Method | URL |
|---------------------|--------|-----|
| Produce a message | `GET` | `http://localhost:8081/kafka/produce?message=Hello%20Kafka` |
| List consumed msgs | `GET` | `http://localhost:8081/kafka/messages` |

### Suggested flow

1. Open `http://localhost:8081/kafka/messages` — the list is empty until something is produced.
2. Produce a message:
```
http://localhost:8081/kafka/produce?message=Message%20sent%20by%20my%20App!
```
3. Call `/kafka/messages` again — you should see the consumed payload(s).

You can use a browser or Postman for both requests.

## How it works

1. **Producer** — `GET /kafka/produce` builds a `User` (`name` + `message`) and sends it to `myTopic` via `KafkaTemplate`.
2. **Consumer** — `MyTopicConsumer` listens on `myTopic` (group `kafka-sandbox`) and stores each message in memory.
3. **Read API** — `GET /kafka/messages` returns that in-memory list.

Broker address is configured in code as `localhost:9092` (`ProducerConfiguration` / `ConsumerConfiguration`).

## License

This project is provided as a learning sample. Feel free to fork and adapt it.
Loading