This repo is a demo for an ElasticSearch indexing daemon that is triggered by a RabbitMQ message. The components used in this example are as follows:
- A RabbitMQ server that acts as the message broker
- An ElasticSearch server
- A Node.js
send.js
script that sends a message containing an AWS S3keyName
,indexName
for the new ElasticSearch index, andindexType
. - A Node.js
receive.js
script that receives the message and does the following:
- Converts the
keyName
CSV file in AWS S3 to a file stream - Parse the file stream and indexes the file into ElasticSearch
You will need 4 terminal windows to run this demo. The demo also assumes that you have installed ElasticSearch, RabbitMQ, and Node.js.
Run the ElasticSearch server
elasticsearch
Run the RabbitMQ server
rabbitmq-server
- Set up the
.env
file with your AWS credentials and S3 bucket name - Install the npm packages and run
receive.js
npm install
node receive.js
node send.js