Kafka DEV

Single-node Apache Kafka (KRaft) for local development via Docker Compose (Windows, Linux, or macOS).

Image: confluentinc/cp-kafka (default 7.9.5). Combined broker + controller — dev only, not for production.

Prerequisites

• Docker Desktop (or compatible Docker Engine + Compose)
• Java / keytool on PATH (to generate TLS material)
• PowerShell (Windows) or bash (Linux / macOS) for the cert script

Quick start

Windows

powershell -ExecutionPolicy Bypass -File .\scripts\generate-certs.ps1
docker compose up -d
docker inspect --format "{{.State.Health.Status}}" kafka-dev

Linux / macOS

chmod +x ./scripts/generate-certs.sh   # once
./scripts/generate-certs.sh
docker compose up -d
docker inspect --format '{{.State.Health.Status}}' kafka-dev

Stop / remove:

docker compose down
# Optional: wipe persisted topics/offsets
rm -rf ./kafka-data

On Windows PowerShell, use Remove-Item -Recurse -Force .\kafka-data instead of rm -rf.

Ports and listeners

Listener	Host port	Protocol	Client auth	Typical bootstrap.servers
EXTERNAL	9092	PLAINTEXT	none	localhost:9092
SSL	9093	TLS	none	localhost:9093
MTLS	9094	TLS	required	localhost:9094
INTERNAL	(docker)	PLAINTEXT	none	kafka:29092 (Compose net)
CONTROLLER	(docker)	PLAINTEXT	—	KRaft only

• From the host: use localhost:9092|9093|9094.
• From another container on the same Compose network: use kafka:29092 (PLAINTEXT internal).

Hostname verification is disabled on the broker (ssl.endpoint.identification.algorithm empty). Client sample configs do the same.

Repository layout

kafka-dev/
├── docker-compose.yml          # Broker service definition
├── scripts/
│   ├── generate-certs.ps1      # Windows: CA, broker, and client stores
│   └── generate-certs.sh       # Linux/macOS: same output
├── docs/
│   └── generate-certs.md       # Cert generator documentation
├── secrets/                    # TLS material (gitignored) — generated locally
├── kafka-data/                 # Broker log dirs / metadata (gitignored)
└── README.md

kafka-data/ (data repository)

• Bind-mounted to /var/lib/kafka/data in the container.
• Holds topic partitions, KRaft metadata, and offsets.
• Gitignored — local only; grows with traffic.
• Delete this folder (with the container stopped) for a clean cluster. Keep the same CLUSTER_ID in docker-compose.yml unless you intentionally re-bootstrap.

secrets/ (certificates)

• Bind-mounted read-only to /etc/kafka/secrets.
• Gitignored — do not commit keystores or password files.
• Regenerated by scripts/generate-certs.ps1 or scripts/generate-certs.sh (scripts are safe to commit).

Certificates

Details for the generator: docs/generate-certs.md.

The scripts build a local self-signed PKI:

Artifact	Purpose
ca.keystore.jks / ca.crt	Dev CA (signing broker + client certs)
kafka.keystore.jks	Broker identity (CN=kafka, SAN: kafka, localhost, 127.0.0.1)
kafka.truststore.jks	Broker trust (CA) — validates mTLS clients
client.keystore.jks	Sample client identity for mTLS
client.truststore.jks	Client trust (CA) — trusts the broker
keystore_creds, key_creds, truststore_creds	Password files Confluent reads at startup
ssl-client.properties	Sample client config for port 9093
mtls-client.properties	Sample client config for port 9094

Default store/key password: kafkadev (dev only).

Regenerate (wipes everything under secrets/ first):

# Windows
powershell -ExecutionPolicy Bypass -File .\scripts\generate-certs.ps1
docker compose up -d --force-recreate

# Linux / macOS
./scripts/generate-certs.sh
docker compose up -d --force-recreate

Creds files

Confluent’s image expects password files next to the JKS stores (not only env passwords):

File	Env in Compose	Unlocks
keystore_creds	KAFKA_SSL_KEYSTORE_CREDENTIALS	kafka.keystore.jks
key_creds	KAFKA_SSL_KEY_CREDENTIALS	Private key inside the keystore
truststore_creds	KAFKA_SSL_TRUSTSTORE_CREDENTIALS	kafka.truststore.jks

Security posture (local)

• Treat secrets/ as disposable laptop material; never promote this CA to staging/prod.
• PLAINTEXT on 9092 is open on the host loopback for convenience.
• mTLS on 9094 proves client identity; it does not replace ACLs/SASL authorization.

Client examples

PLAINTEXT

bootstrap.servers=localhost:9092

SSL (encryption only)

docker exec kafka-dev kafka-topics --bootstrap-server localhost:9093 `
  --command-config /etc/kafka/secrets/ssl-client.properties --list

mTLS

docker exec kafka-dev kafka-topics --bootstrap-server localhost:9094 `
  --command-config /etc/kafka/secrets/mtls-client.properties --list

Copy client.keystore.jks and client.truststore.jks into other local tools as needed (password kafkadev, type JKS).

Useful commands

docker compose logs -f kafka
docker compose ps
docker exec kafka-dev kafka-topics --bootstrap-server localhost:29092 --list

Override image version:

$env:CONFLUENT_VERSION = "7.9.5"
docker compose up -d

Defaults worth knowing

• Auto-create topics: on
• Default partitions: 3, replication factor: 1
• Heap: -Xms512M -Xmx1G
• Cluster ID (fixed in Compose): MkU3OEVBNTcwNTJENDM2Qk