RabbitMQ -> Amazon forwarder

Application to forward messages from RabbitMQ to different Amazon services.

Key features:

forwarding RabbitMQ message to AWS SNS topic
forwarding RabbitMQ message to AWS SNS queue
triggering AWS lambda function directly from RabbitMQ message
automatic RabbitMQ reconnect
message delivery assurance based on RabbitMQ persistency and AWS error handling
dedicated dead-letter exchange and queue creation
http health checks and restart functionality

Architecture

Configuration

The list of RabbitMQ sources and corresponding AWS target resources are stored in mapping file.

Mapping file

Sample of RabbitMQ -> SNS mapping file. All fields are required. Samples are located in examples directory.

[
  {
    "source" : {
      "type" : "RabbitMQ",
      "name" : "test-rabbit",
      "connection" : "amqp://guest:guest@localhost:5672/",
      "topic" : "amq.topic",
      "queue" : "test-queue",
      "routingKeys" : ["#"]
    },
    "destination" : {
      "type" : "SNS",
      "name" : "test-sns",
      "target" : "arn:aws:sns:eu-west-1:XXXXXXXX:test-forwarder"
    }
  }
]

Environment variables

Forwarder uses the following environment variables:

export MAPPING_FILE=/config/mapping.json
export AWS_REGION=region
export AWS_ACCESS_KEY_ID=access_key
export AWS_SECRET_ACCESS_KEY=secret_key

Using TLS with rabbit

Specify amqps for the rabbit connection ub the mapping file:

 "connection" : "amqps://guest:guest@localhost:5671/",

Additional environment variables for working with TLS and rabbit:

export CA_CERT=/certs/ca_certificate.pem
export CERT_FILE=/certs/client_certificate.pem
export KEY_FILE=/certs/client_key.pem

Amazon configuration

When making subscription to SNS -> SQS/HTTP/HTTPS set Raw message delivery to ensure that json messages are not escaped.

Build docker image

make build

Run

Using docker:

docker run \
-e AWS_REGION=$AWS_REGION \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-e MAPPING_FILE=/config/mapping.json \
-v $MAPPING_FILE:/config/mapping.json \
-p 8080:8080 \
airhelp/rabbit-amazon-forwarder

Using docker-compose:

docker-compose up

Test

docker-compose build --pull
docker-compose run --rm tests

Release

make push
docker tag airhelp/rabbit-amazon-forwarder airhelp/rabbit-amazon-forwarder:$VERSION
docker push airhelp/rabbit-amazon-forwarder:$VERSION

Supervisor

Supervisor is a module which starts the consumer->forwarder pairs. Exposed endpoints:

APP_URL/health - returns status if all consumers are running
APP_URL/restart - restarts all consumer->forwarder pairs

AirHelp/rabbit-amazon-forwarder