Spring Boot project reading events from Live Objects MQTT and pushing them to Azure Event Hub
This project is intended for Live Objects users wishing to explore integration patterns with Azure and for organizations already running business logic on Azure planning to work on events from IoT devices sourced via Live Objects.
It is assumed that the reader of this document is familiar with Live Objects and Azure. In order to run the connector, it is required to have access to:
- a Live Objects account
- with MQTT fifo queue
- API key which can access the queue (API key generation is described in the user guide)
- an Azure account
- Event Hub set up (creation process is described in official documentation)
- Primary Key to event hub (it is automatically generated by Azure, see
all resources -> your event hubs namespace -> Shared access policies)
- computer with
- Java-supporting IDE (e.g. IntelliJ, Eclipse)
- Azure CLI (https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest)
- Apache Maven
- Git client
The connector (Mqtt2EvtHub) subscribes to selected Live Objects MQTT queue, reads all events and publishes them to selected Azure Event Hub without any modification to events’ contents. It is intended to be run as a long-running process hosted on Azure. Connector code is written in Java, using the Spring Boot framework.
Mqtt2EvtHub supports only the communication from Live Objects i.e. it reads messages sent from IoT devices. Communication towards devices (sending commands to devices) is not supported.
Provisioning of IoT devices is within the scope of Live Objects; Mqtt2EvtHub has no knowledge on what devices are communicating; it is just transparently moving messages from Live Objects to Azure Event Hub. It is assumed that business logic acting on those messages is to be applied by applications consuming the messages from Event Hub.
The software is an open source toolbox which has to be integrated into an end to end solution. Mqtt2EvtHub comes without any guarantees related to percentage of messages successfully written to Event Hub, nor to the response time. Moreover, the ordering of messages is not guaranteed to be preserved; the application uses thread pools to run its MQTT and Event Hub adapters which may cause some messages to arrive in Event Hub out of order in which they were kept within Live Objects’ MQTT queue.
Tested on Azure P2v2 App Service Plan, the connector was processing events at the rate of ~120,000 messages per minute (please keep in mind that Event Hub has a configurable throughput limit, with single throughput unit corresponding to ~60,000 messages per minute). This should not be treated as a guarantee either, since the throughput depends i.a. on message size and number of applications shared within App Service Plan.
Live Objects platform supports load balancing between multiple MQTT subscribers. It is possible to run multiple instances of Mqtt2EvtHub, each of them will handle its own subset of messages.
In order to be deployed to Azure, the project uses Azure Webapp Maven Plugin. The installation description is based on the following tutorial: https://docs.microsoft.com/en-us/java/azure/spring-framework/deploy-spring-boot-java-app-with-maven-plugin You can refer to it in case of troubles with current project, or if you want to better understand how it works.
Application requires an Application Service Plan to run on Azure. App Service Plan defines compute resources available for the application during execution, exposing a runtime platform (Platform as a Service). It is a paid service, the price depending on CPU and RAM configuration.
To create a Plan, enter Azure Web Portal and select “create a resource” option. In the search field type “App Service Plan”, click “Create”. Now you need to provide a name, operating system (connector was tested on Linux, but it should work without problems on Windows as well), location, resource group and the size. Afterwards, click “Review and create”, then final “Create”.
Application is using standard Spring mechanisms for configuration, with all properties stored within application.yml file. Properties values can be kept there and/or overridden from other sources (e.g. from environment variables) – see more information in Spring documentation. One of possible methods for setting Spring app configuration parameters from Azure is to define JAVA_OPTS environment variable and place configuration values in it. For example, in order to specify lo-mqtt.api-key and lo-mqtt.topic, one would need to set JAVA_OPTS to:
-Dlo.api-key=apikey –Dlo.topic=fifo/topic
More information on JAVA_OPTS in Azure context can be found here.
The following properties are used for integration:
URI of the Live Objects platform.
Default value liveobjects.orange-business.com
API key to authorize the MQTT connector. Information on how to create a Live Objects API key can be found in Live Objects documentation.
There is no sensible default value since it should correspond to your account on Live Objects.
Connector uses Application mode to connect to Live Objects Platform so username is internally set to application and it cannot be changed.
Name of the MQTT queue. For example, if Live Objects web portal displays a FIFO named “dev” (seen under “Data -> FIFO” option), to subscribe to this queue the property should be set to dev.
Namespace in which your Event Hub is created.
Name of Event Hub to which the MQTT messages should be written to.
It is a policy name. The default one, created by Azure, is named RootManageSharedAccessKey. You can verify the name and key value in the Azure portal, by checking all resources -> your event hubs namespace -> Shared access policies.
This is a Primary key from the policy above.
Now, get this github project on your development machine, using for instance:
git clone https://github.com/DatavenueLiveObjects/Azure-Event-Hub-connector-to-install-on-Azure.git
Deployment to Azure is performed by the Azure Webapp Maven Plugin. Its configuration is included in pom.xml file within the connector project.
The following lines are relevant:
<resourceGroup>JavaApps</resourceGroup>
<appServicePlanName>PremiumV2Plan</appServicePlanName>
<appName>lo-event-hub</appName>
The resourceGroup and appServicePlanName should correspond to values provided during App Service Plan creation. appName will be used to uniquely identify the deployed connector app. Please keep in mind that appName must be unique application name in whole azurewebsites.net subdomain to avoid dns collision.
It is also possible to use one of the releases at https://github.com/DatavenueLiveObjects/Azure-Event-Hub-connector-to-install-on-Azure/releases by manually executing the application on a Linux VM.
Azure login
In order to deploy the application, the prerequisite is to have a logged session to Azure. Run the following command using Azure CLI tool:
az login
Follow the instructions to complete the login process.
JAR deployment
Build the JAR file using command:
mvn clean package
Deploy the application with the command:
mvn azure-webapp:deploy
If you repeat those steps, application will be redeployed, replacing the previously-deployed instance on Azure. Please keep in mind that you don’t have to repeat the login to Azure (unless the session expired, which will result in appropriate error message during app deployment).
To check if connector is deployed properly you can:
- go under
your_webapp_url/actuator/to see some useful information, especially underyour_webapp_url/actuator/health - go to the web app log stream in Azure Portal - , menu “Monitoring”, sub-menu “Log Stream”, to see the stdout of the application
- go to EventHub overview to see few charts with (among others) incoming and outcoming messages
If wrong configuration to EventHub is given, Azure connector will throw a com.microsoft.azure.eventhubs.EventHubException wrapping a java.lang.NullPointerException. In such case, make sure that you provide proper EventHub namespace, hub name, SAS key name and value.