/Study21

Primary LanguageJava

Study21 README Contents

Research the Pulsar message broker

Apache Pulsar is a distributed messaging and streaming platform. The Pulsar terminology.

This application uses classes from the pulsar-client-api library:

  • PulsarClient
  • Producer
  • Consumer
  • Reader
  • Message

The sections of this project:

  1. Docker Build
  2. Originators, Selectors, Destinations
  3. Reporter with Information Checker and Message Displayer
  4. Docker and Pulsar Scripts

Java source code. Packages:

application sources :  kp
test sources :  kp


Java API Documentation


❶ Docker Build

Action:

1. With batch file "01 Docker Pulsar.bat" install and start the container with the Pulsar.
2. With batch file "02 Pulsar create tenant and namespace.bat"> create the tenant and the namespace.
3. With batch file "03 Docker build and run.bat"> build the image and
start the container with the application.

1.1. Docker image is built using these files: Dockerfile and compose.yaml.


The message flows between the containers in Docker.

The screenshot of the created Docker containers.

1.2. The integration test for the message exchange between 'KpOriginator', 'KpSelector', and 'KpDestination'.
It uses the 'PulsarContainer'. This is the Apache Pulsar Module from Testcontainers for Java.

Back to the top of the page


❷ Originators, Selectors, Destinations

2.1. All received messages are acknowledged. For the exceptions it is used the negative acknowledge.
For the payload send in Pulsar 'Message' it is used the 'Information' object. The 'Information' contains the list of the 'Department' objects. The 'Department' contains the list of the 'Employee' objects.


The flowchart diagram for two origins, destination selector, origin selector, and two destinations.


The sequence diagram for two origins, destination selector, origin selector, and two destinations.

2.2. It is implemented the fail-over logic. There are two identical Docker containers: osd-selector-dest-a and
osd-selector-dest-b. They have the consumer subscribed with the Fail-over subscription type. Pulsar configures these two containers and makes one consumer active and thr other consumer on standby.

The fail-over test steps:

  • container with the active consumer is manually stopped in Docker
  • it causes the active consumer to be disconnected
  • Pulsar makes the standby consumer active
  • this activated by Pulsar consumer starts consuming messages

2.3. The producers in the container osd-originator-1 and osd-originator-2 produce messages to the topic 'select-dest'.
The producer method: kp.originators.KpOriginator::sendMessage.


The log from the container osd-originator-1.

2.4. The destination selectors in the containers osd-selector-dest-a and osd-selector-dest-b direct the messages from the destination selector topic 'select-dest' to the destination topics 'dest-1' and 'dest-2'.
In these two selectors Pulsar makes one consumer active and the other consumer on standby.

The message selection logic:

  • a message with odd information id goes to 'dest-1'
  • a message with even information id goes to 'dest-2'

The consumer and the producer method: kp.selectors.KpSelector::select.


The log from the container osd-selector-dest-a.

2.5. The consumer in the container osd-destination-1 consumes messages from the topic 'dest-1'.
The consumer in the container osd-destination-2 consumes messages from the topic 'dest-2'.
They send the response to the topic 'select-orig'.
Received 'Information' is approved or rejected (logic with flip-flop sequence).

The consumer method: kp.destinations.KpDestination::receiveAndRespond.


The log from the container osd-destination-1.

2.6. The origin selector in the container osd-selector-orig directs the messages from the origin selector topic 'select-orig' to the origin topics 'orig-1' and 'orig-2'.

The message selection logic:

  • messages with odd information id go to 'orig-1'
  • messages with even information id go to 'orig-2'


The log from the container osd-selector-orig.

2.7. The consumer in the container osd-originator-1 consumes messages from the topic 'orig-1.
The consumer in the container osd-originator-2 consumes messages from the topic 'orig-2.

The consumer method: kp.originators.KpOriginator::receive.

The relevant log from the container osd-originator-1 is presented in point 2.1 above.

Back to the top of the page


❸ Reporter with Information Checker and Message Displayer

The reporter uses the Pulsar Reader.

Action:

1. With batch file "04 Docker start reporter.bat" start the reporter.


The reporter diagram

3.1. The reporter checks the 'Information' in two messages. One message is from the destination topic and the other message is from the origin topic.

  1. searches for the Pulsar messages in the topic 'select-dest'
  2. for checking, it selects the message from the middle of the received list
  3. this is the source of expected 'Information'
  4. searches for the related message in topics 'orig-1' and 'orig-2'
  5. this is the source of actual 'Information'
  6. compares the expected 'Information' with the actual 'Information'
  7. the 'Information' in both messages should be equal

The searching and checking method: kp.reporters.KpReporter::findAndCheck.

The screenshot of the log from the container osd-reporter.

3.2. The reporter displays the latest message.

  1. in topic 'dest-1' or topic 'dest-2' it finds the latest message
  2. displays the 'Information'

The searching and displaying method: kp.reporters.KpReporter::findLatestAndDisplay.

The screenshot of the log from the container osd-reporter.

Back to the top of the page


❹ Docker and Pulsar Scripts

Action:

1. With batch file "05 Docker list.bat" run the script for Docker to show the network and the containers.
2. With batch file "06 Pulsar list.bat" run the pulsar-admin to show the tenant, the namespace, the topics, and
the subscription.

4.1. The batch file "05 Docker list.bat".
The screenshot of the console log from the run of the batch file "05 Docker list.bat".

4.2. The batch file "06 Pulsar list.bat".
The screenshot of the console log from the run of the batch file "06 Pulsar list.bat".

Back to the top of the page