Overview

This is a sample implementation of trigger monitor for IBM MQ and AWS Lambda, together with configuration, scripts and data to run a demo. The sample is designed to provide illustration and guidance for developing Lambda functions in Java that are triggered by MQ Lambda Trigger Monitor, as well as samples of configuring these functions and related resources on AWS.

The repository includes the following components:

Lambda function sample - A Java project implementing AWS Lambda function, triggered when messages arrive on a specially configured MQ queue. The Lambda function parses the messages and inserts corresponding records into a MySQL database. The project consists of two modules:
Generic message handler - can be used as is for any MQ triggered Lambda. It implements all MQ-related queue and message manipulation logic. It provides an abstract class that needs to be extended for any application-specific message processing functionality such as database manipulation, SQS or Kinesis message propagation etc.
Sample message processor - a sample usage of generic message handler that parses input messages and inserts data into a MySQL database.
Demo - subcomponents to prepare and run a demo of the sample. It includes a script that builds the sample project, uploads it to AWS S3, and then creates a CloudFormation stack with all the necessary resources to run the demo.

Demo description

The demo includes a CloudFormation template that creates all the resources necessary for running the sample end to end:

An EC2 instance that hosts MQ Queue Manager along with MQ Lambda Trigger Monitor
A fully configured Lambda function, triggered by MQ Lamdba Trigger Monitor, pulls messages from the triggered queue and writes records to a sample database
An RDS MariaDB instance with a sample database
All auxiliary resources, including a VPC instance, subnets, security groups, internet gateway and IAM roles.

Note: The Lambda function in the sample template is configured with a very short timeout of 10 seconds. This is done only to make sure the function does not remain up and contributes to billable time. In production usage, you need to set the timeout according to the message arrival patterns. For queues that are very busy and have constant traffic you probably want much longer timeouts. When Lambda function times out, MQ will automatically retrigger it immediately if there are still messages on the queue. The sample message handler monitors time remaining before time out, and terminates gracefully 1 second in advance, releasing MQ resources and thus making sure retriggering occurs instantly.

The EC2 instance is configured with a cloud-init script, which runs the sample:

Builds a Docker image with base IBM MQ Server, using IBM MQ Advanced for Developers. The Dockerfile is pulled from IBM MQ Docker repository
Pulls mqlambdatm release
Builds a custom image based on mq-docker, which includes MQ trigger configuration and mqlambdatm executable
Creates an empy sample database
Runs Docker container with MQ Queue Manager and mqlambdatm as a Queue Manager-controlled service
Executes MQ sample amqsput to populate the triggered queue with sample data
Waits for the triggered queue to become empty as a result of Lambda function processing all messages
Displays the row counts for sample tables

Running demo

You need to install and configure AWS CLI. Make sure the IAM credentials you configure with AWS CLI can create CloudFormation stacks with EC2, RDS, Lamda and IAM resources. Make also sure to configure default AWS region.

Additionally, you need:

JSON command line processor jq
Apache Maven
git
unzip

Depending on your distribution:

sudo apt-get install jq maven git unzip

sudo yum install jq maven git unzip

You need IBM MQ Java client library, com.ibm.mq.allclient.jar, If you have IBM MQ with Java option installed, it is in the directory <MQ_INSTALLATION_PATH>/java/lib, by default /opt/mqm/java/lib. If you don't have MQ installed on your computer, then you need to download IBM MQ redistributable client for Java. Follow this link, scroll down and click on the link above "IBM MQ JMS and Java redistributable client". You will need IBMid and password to proceed, if you don't have one you need to create it from this page.

After downloading, unzip the file into any directory.

Then:

git clone https://github.com/yurist/mqlambdatm-demo.git
cd mqlambdatm-demo/demo
./demo.sh --mq-jar-dir <absolute path to mq java client jar directory> --key-pair <your existing key pair> --deployment-bucket <existing S3 bucket>

The scripts reports its progress while executing. The last step, waiting for the stack creation to complete, takes between 5 to 10 minutes. Most of this time is spent initializing the RDS instance. The demo is optimized to run the IBM MQ installation simultaneously with RDS instance creation, but it still takes a while to complete.

You can ssh to the EC2 instance after the script finishes to check the cloud-init logs at /var/log/cloud-init-output.log and see the demo progress.

You can remove all demo resources by deleting the stack created by the script. The script displays the relevant instructions.

Use of IBM software

The demo downloads and installs IBM MQ Advanced for Developers. By running the demo, you accept IBM License agreement

License

Apache License, Version 2.0