Dockerizing Spring Boot Application & Kafka Broker

Pre-Requisites

In your OS or VM:

Install docker
Install docker-compose

This demo uses Ubuntu Linux (16.04 LTS version) in Virtual Box.

Introduction

The purpose of the repository is to showcase how to dockerize a Spring Boot application as well as a Kafka Broker (along with Zookeeper) on either a Single Host Server or on a Swarm Cluster.

Docker Architectural Layers

Stack (Swarm Cluster)
Services (Single Host)
Containers (The lowest layer)

We are focusing on "Services" and "Swarm" layers for this demo.

Use Cases

We know dockerization can be used for:

Development environment
Automated testing environment
Single host deployment
Swarm cluster deployment

Swarm Features

On a Swarm cluster, the Swarm mode features:

Scaling
Load balancing
Decentralized design
Multi-host networking
Declarative service model
Service discovery
Rolling updates
etc.

The Spring Boot application uses Spring Cloud Stream to communicate (publish/receive messages) with Kafka message broker. The application simulates a compute engine to perform simple calculating tasks generated by REST 'send' request, and return the result to the user by REST 'receive' request. It uses Kafka topics to store requests and responses.

Preparation: Docker Images

To facilitate the demostration of dockerization, I created two docker images:

Also it's worth noting how Kafka (Zookeeper) docker image is built:

Kafka Docker Repo @ Github

Both docker images are used to create multiple docker containers, which are managed through docker-compose. The docker-compose prescribes the docker stack, services, networks, and deployment, etc.

Pulling Docker Images from Docker Hub to Local Repository

docker pull michaeldqin/springboot-kafka

docker pull michaeldqin/kafka

Running the Spring Boot Application and Kafka Broker on a Single Host Server

In order to run the docker containers, we need a docker-compose.yml to orchestrate the services.

Docker-Compose-Single-Host File

File: docker-compose-single-host.yml

version: '3'
services:
  springboot-webapp:
    image: michaeldqin/springboot-kafka
    ports:
    - 8080:8080
    depends_on:
    - kafka-server
    networks:
    - springboot-network
    logging:
      driver: json-file
  kafka-server:
    image: michaeldqin/kafka
    hostname: kafka-server
    ports:
    - "2181:2181"
    - "9092:9092"
    environment:
    - ADVERTISED_LISTENERS=PLAINTEXT://kafka-server:9092    
    volumes:
    - ${PWD}/.:/opt/kafka_2.12-1.0.0/log
    networks:
    - springboot-network
    logging:
      driver: json-file
networks:
  springboot-network:
    driver: bridge

Start the Application Services on a Single Host Server

docker-compose -p dev -f docker-compose-single-host.yml up

Please note, that dev is the project name, and it serves as the development environment for developers.

Check the Running Containers

docker ps

Open Spring Boot App Home

In browser, type: http://localhost:8080

Test Spring Boot App RESTful Web Services

In browser, type: http://localhost:8080/send

In browser, type: http://localhost:8080/receive

Take Down the Application Services

docker-compose -p dev -f docker-compose-single-host.yml down

Running the Spring Boot Application and Kafka Broker on a Swarm Cluster

In a Swarm cluster mode, we can have 3 application instances running, and 1 Kafka broker serving all the messages.

Docker-Compose-Swarm-Cluster File

File: docker-compose-swarm-cluster.yml

version: '3'
services:
  springboot-webapp:
    image: michaeldqin/springboot-kafka
    ports:
    - 8080:8080
    depends_on:
    - kafka-server
    networks:
    - webnet
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
  kafka-server:
    image: michaeldqin/kafka
    hostname: kafka-server
    ports:
    - "2181:2181"
    - "9092:9092"
    environment:
    - ADVERTISED_LISTENERS=PLAINTEXT://kafka-server:9092    
    volumes:
    - ${PWD}/.:/opt/kafka_2.12-1.0.0/log
    networks:
    - webnet
    deploy:
      placement:
       constraints: [node.role == manager]
networks:
  webnet:

Please note, the only difference between docker-compose-single-host.yml and docker-compose-swarm-cluster.yml is the deployment instructions in the docker-compose-swarm-cluster.yml.

Start the Application Stack on a Swarm Cluster

# Initialize a Swarm cluster
docker swarm init

# Deploy the stack as 'dev' environment
docker stack deploy -c docker-compose-swarm-cluster.yml dev

docker swarm init

docker stack deploy

Check the Status of the Running Services on a Swarm Cluster

docker service ls
## OR
docker container ls

## Check dev_springboot-webapp instances
docker service ps dev_springboot-webapp

Testing Load Balancing

Open your favorite browser, in address bar you type: http://localhost:8080, then enter key. You can see Hostname is different from different browser windows, which means multiple application instances are serving the requests.

Testing Scaling

Update Docker-Compose File

You can scale the app by changing the replicas value in docker-compose.yml, saving the change, and re-running the docker stack deploy command.

Update: replicas: 5

services:
  springboot-webapp:
    ...
    deploy:
      replicas: 5
    ...

Re-Run:

docker stack deploy -c docker-compose-swarm-cluster.yml dev

Take Down the Application Stack and the Swarm Cluster

# Take down the stack (application)
docker stack rm dev

# Take down the Swarm cluster
docker swarm leave --force

# Remove containers
docker container rm $(docker container ls -a -q)

Docker CLI Reference

docker stack ls                                            # List stacks or apps
docker stack deploy -c <composefile> <appname>  # Run the specified Compose file
docker service ls                 # List running services associated with an app
docker service ps <service>                  # List tasks associated with an app
docker inspect <task or container>                   # Inspect task or container
docker container ls -q                                      # List container IDs
docker stack rm <appname>                             # Tear down an application
docker swarm init             # Create a single node (manager) swarm and join it
docker swarm leave --force      # Take down a single node swarm from the manager

MikeQin/dockerizing-springboot-kafka