Apache Pulsar is a distributed messaging and streaming platform.
The Pulsar terminology.
This application uses classes from the pulsar-client-api library:
- PulsarClient
- Producer
- Consumer
- Reader
- Message
The sections of this project:
- Docker Build
- Originators, Selectors, Destinations
- Reporter with Information Checker and Message Displayer
- Docker and Pulsar Scripts
Java source code. Packages:
application sources :
kp
test sources :
kp
Action:
1. With batch file
"01 Docker Pulsar.bat" install and start the container with the Pulsar.
2. With batch file
"02 Pulsar create tenant and namespace.bat"> create the tenant and the namespace.
3. With batch file
"03 Docker build and run.bat"> build the image and
start the container with the application.
1.1. Docker image is built using these files:
Dockerfile and
compose.yaml.
The message flows between the containers in Docker.
The screenshot of the created Docker containers.
1.2. The integration test for the message exchange between 'KpOriginator', 'KpSelector', and 'KpDestination'.
It uses the 'PulsarContainer'.
This is the Apache Pulsar Module from
Testcontainers for Java.
2.1. All received messages are acknowledged. For the exceptions it is used the negative acknowledge.
For the payload send in Pulsar 'Message' it is used the 'Information' object.
The 'Information' contains the list of the 'Department' objects.
The 'Department' contains the list of the 'Employee' objects.
The flowchart diagram for two origins, destination selector, origin selector, and two destinations.
The sequence diagram for two origins, destination selector, origin selector, and two destinations.
2.2. It is implemented the fail-over logic.
There are two identical Docker containers: osd-selector-dest-a and
osd-selector-dest-b. They have the consumer subscribed with the Fail-over subscription type.
Pulsar configures these two containers and makes one consumer active and thr other consumer on standby.
The fail-over test steps:
- container with the active consumer is manually stopped in Docker
- it causes the active consumer to be disconnected
- Pulsar makes the standby consumer active
- this activated by Pulsar consumer starts consuming messages
2.3. The producers in the container osd-originator-1 and osd-originator-2 produce messages to the topic 'select-dest'.
The producer method:
kp.originators.KpOriginator::sendMessage.
The log from the container osd-originator-1.
2.4. The destination selectors in the containers osd-selector-dest-a and osd-selector-dest-b
direct the messages from the destination selector topic 'select-dest' to the destination topics 'dest-1' and 'dest-2'.
In these two selectors Pulsar makes one consumer active and the other consumer on standby.
The message selection logic:
- a message with odd information id goes to 'dest-1'
- a message with even information id goes to 'dest-2'
The consumer and the producer method: kp.selectors.KpSelector::select.
The log from the container osd-selector-dest-a.
2.5. The consumer in the container osd-destination-1 consumes messages from the topic 'dest-1'.
The consumer in the container osd-destination-2 consumes messages from the topic 'dest-2'.
They send the response to the topic 'select-orig'.
Received 'Information' is approved or rejected (logic with flip-flop sequence).
The consumer method: kp.destinations.KpDestination::receiveAndRespond.
The log from the container osd-destination-1.
2.6. The origin selector in the container osd-selector-orig directs
the messages from the origin selector topic 'select-orig' to the origin topics 'orig-1' and 'orig-2'.
The message selection logic:
- messages with odd information id go to 'orig-1'
- messages with even information id go to 'orig-2'
The log from the container osd-selector-orig.
2.7. The consumer in the container osd-originator-1 consumes messages from the topic 'orig-1.
The consumer in the container osd-originator-2 consumes messages from the topic 'orig-2.
The consumer method: kp.originators.KpOriginator::receive.
The relevant log from the container osd-originator-1 is presented in point 2.1 above.
The reporter uses the Pulsar Reader.
Action:
1. With batch file
"04 Docker start reporter.bat" start the reporter.
The reporter diagram
3.1. The reporter checks the 'Information' in two messages. One message is from the destination topic and the other message is from the origin topic.
- searches for the Pulsar messages in the topic 'select-dest'
- for checking, it selects the message from the middle of the received list
- this is the source of expected 'Information'
- searches for the related message in topics 'orig-1' and 'orig-2'
- this is the source of actual 'Information'
- compares the expected 'Information' with the actual 'Information'
- the 'Information' in both messages should be equal
The searching and checking method: kp.reporters.KpReporter::findAndCheck.
The screenshot of the log from the container osd-reporter.
3.2. The reporter displays the latest message.
- in topic 'dest-1' or topic 'dest-2' it finds the latest message
- displays the 'Information'
The searching and displaying method: kp.reporters.KpReporter::findLatestAndDisplay.
The screenshot of the log from the container osd-reporter.
Action:
1. With batch file
"05 Docker list.bat" run the script for Docker to show the network and the containers.
2. With batch file
"06 Pulsar list.bat" run the pulsar-admin to show the tenant, the namespace, the topics, and
the subscription.
4.1. The batch file "05 Docker list.bat".
The
screenshot of the console log from the run of the batch file "05 Docker list.bat".
4.2. The batch file "06 Pulsar list.bat".
The
screenshot of the console log from the run of the batch file "06 Pulsar list.bat".