This project demonstrates how to use Red Hat AMQ + Red Hat Data Grid and Red Hat Build of Camel with Quarkus to create a data layer with a canonical data model.
Below is a description of each component of the solution:
- camel-amq-datagrid-common:
- Contains the canonical model of the data that will be available in the Data Grid.
- This project is shared between Camel transformation routes and Data Grid customers.
- Transformation layer
- app1-camel-amq-datagrid:
- Responsible for receiving notifications about events generated by app1 in the queue APP1_DATA_RECORD_QUEUE and update the data in the Data Grid.
- The route converts the model data known only by App1 in json format to the canonical model defined in camel-amq-datagrid-common.
- Monitors topic CACHE_UPDATE_REQUEST for cache refresh request. Triggers a REST API of the service app1 requesting to add updated data to the message queue APP1_DATA_RECORD_QUEUE
- app2-camel-amq-datagrid:
- Responsible for receiving notifications about events generated by app2 in the queue APP2_DATA_RECORD_QUEUE and update the data in the Data Grid.
- The route converts the model data known only by App2 in xml format to the canonical model defined in camel-amq-datagrid-common.
- Monitors topic CACHE_UPDATE_REQUEST for cache refresh request. Triggers a REST API of the service app2 requesting to add updated data to the message queue APP2_DATA_RECORD_QUEUE
- app1-camel-amq-datagrid:
- Access layer
- front-rest-cache-client:
- Demonstrates how to consume data directly from the Data Grid and expose it through a REST API and an HTML interface. If the record is not found in the Data Grid, the application publishes a message in the CACHE_UPDATE_REQUEST topic requesting that the applications publish data about the record in the AMQ MESSAGE BROKER and thus the Data Grid is updated.
- Demonstrates how to configure and consume a Data Grid event listener to update the GUI in real time using Server-Sent Events (SSE)
- Expose API in URL
http://localhost:8090/api
- app-camel-rest-cache:
- Exposes the record query through a REST API simplifying access for clients that don't need to deal with connectivity to the Data Grid.
- If the record is not found in the Data Grid, the application publishes a message in the CACHE_UPDATE_REQUEST topic requesting that the applications publish data about the record in the AMQ MESSAGE BROKER and thus the Data Grid is updated.
- Expose API in URL
http://localhost:8089/api
- nodejs-rest-client:
- A simple client that consumes the API published by the app-camel-rest-cache service and displays it through an HTML interface.
- front-rest-cache-client:
- Producer layer
- app1:
- Represents an application with its particular data model. Publishes data changes to the APP1_DATA_RECORD_QUEUE queue and receives notifications via REST API from app1-camel-amq-datagrid camel route to provide data.
- app2:
- Represents an application with its particular data model. Publishes data changes to the APP2_DATA_RECORD_QUEUE queue and receives notifications via REST API from app2-camel-amq-datagrid camel route to provide data.
- app1:
- JDK 17+
- Docker runtime 24+
- Docker Compose 2.19+
- NPM 9.5+
- Node 18+
Note The examples below were run on a MacOS Ventura 13.4.1
Warning Before running the commands below, check your platform configuration to support the above requirements.
./mvnw clean install -DskipTestsdocker compose down && docker compose upcd app1
./mvnw quarkus:dev -Ddebug=false
cd app2
./mvnw quarkus:dev -Ddebug=falsecd app1-camel-amq-datagrid
./mvnw quarkus:dev -Ddebug=false
cd app2-camel-amq-datagrid
./mvnw quarkus:dev -Ddebug=falsecd app-camel-rest-cache
./mvnw quarkus:dev -Ddebug=falsecd front-rest-cache-client
./mvnw quarkus:dev -Ddebug=falsecd front-nodejs-rest-client
npm install
node server.js./mvnw clean install -Dskiptestsdocker compose -f docker-compose-complete.yml down && docker compose -f docker-compose-complete.yml up --buildNote Before running the tests it is necessary that all services are properly active, use the way you want above.
- Access the front-rest-cache-client service web interface
http://localhost:8090
- Access the Data Grid admin console and navigate to the cache DATA-LAYER-CACHE page
http://localhost:11222/console/cache/DATA-LAYER-CACHEuser: quarkus password: password
Warning At first, the cache will be completely empty. In a productive environment it is necessary to decide if this is the expected behavior or if auxiliary services will be necessary to "warm up" the cache before the applications look for information in it.
Note In this solution, the applications look for the registration key in the cache and if they don't find it, they start a process to populate the cache. In the case of the front-rest-cache-client application, which is configured to listen to events from the Data Grid and has the capacity to receive events from the server, the interface is updated in real time.
-
Access the front-nodejs-rest-client service web interface
http://localhost:3000. The interface is exactly the same as the front-rest-cache-client service -
In the front-rest-cache-client interface, look for record 1. Naturally, this value will not be found in the cache, so the service will send an update request by publishing a message in topic CACHE_UPDATE_REQUEST in the AMQ Broker.
Note We are talking about a topic and not a queue, as this update request must be answered by all applications that supply data to the canonical Data Grid model.
- The first response will be something similar to the snippet below, indicating that an update process for the requested record has been initiated.
{
"type": "LOAD",
"registry": null,
"message": "Data will be loaded, please wait a few seconds"
}Note Depending on the speed of return, it is possible that the message is not even displayed.
- A few seconds later, you will be able to see the interface update with a sample of randomly generated data
Note In a real environment this data would possibly come from a database, for the sake of simplicity, both app1 and app2 make use of templates to generate examples with random data.
-
Go to front-nodejs-rest-client interface and also search for key 1, note that the return will be exactly the same.
-
Still in front-nodejs-rest-client, search for key 2. Note that the key was not found and the service only returns a message.
- Repeating the query, this time it is possible to see the value generated by the apps app1 and app2
The project includes a postman collection , you can import and use it to run the tests below.
- Using the "app1 create provided data" request, it is possible to publish an arbitrary data structure with the values you want in the APP1_DATA_RECORD_QUEUE queue. (surely respecting the data model known by the camel route app1-camel-amq-datagrid).
- Access the Data Grid administrative console and check that a new object in the cache was created with the indicated key (id field of the request body).
Note Records in cache are configured to stay alive for a maximum of 5 minutes
- Go to front-rest-cache-client interface and also search for the same key, note that the return will be exactly the same.
- Using the "app2 create provided data" request, add information referring to the data provided by app2 to the same key. In app2 the data model is based on xml.
- Notice in the front-rest-cache-client interface that the data has been enriched and now contains information from both applications.
- Both "app1 generate random data" and "app2 generate random data" requests generate template-based data with random values. The last part of the URL represents the path parameter that defines the value of the key that will be used in the Data Grid.
http://localhost:8091/message/random/1andhttp://localhost:8092/message/random/1
Note These are exactly the same services invoked when using some interface to fetch a key that does not exist in the cache.