This project is a custom watcher for Elasticsearch which works with Apache Kafka by reacting to event in Elasticserch and writing them to Apache Kafka.
Supports 2 types of triggers.
This trigger uses crons to trigger when an event should be pushed to Apache Kafka based on the watch configuration.
This trigger relies on the IndexListeners and DeleteListeners. Once data is either created or deleted it triggers all watchers that meet criteria and pushes the data to Apache Kafka.
Plugin requires a configuration file to know how to connect to Apache Kafka and also how it would use elasticsearch report engine to generate reports. The file elasticsearch-kafka-watch.yml should be place in /path/to/elasticsearch/config folder to be picked by plugin.
sudo bin/elasticsearch-plugin install [plugin_name]
Create a custom watch with its cron. Events would be generated using the cron. This is written into Apache Kafka. Any worker/consumer listening on Apache Kafka would react to the event.
For sending SMS or Email alerts based on events written in Apache Kafka check out go kafka alert.
Creating a custom watch for a time based cron expects the following parameters:
a.
b.
c.
Create a custom watch with and elasticsearch index and query. Once data is written or deleted from the index, it triggers the custom watch to evaluate query to check if there'll be a hit greater than 0. Once this is positive an event is written to Apache Kafka for consumers/workers listening to react.
For sending SMS or Email alerts based on events written in Apache Kafka check out go kafka alert.
Creating a custom watch for a time based cron expects the following parameters:
a.
b.
c.
This plugin also works with 2 other plugins to schedule reports using elasticsearch as datasource.
elasticsearch report engine and go kafka alert. The former generates PDF,CSV and HTML reports from elasticsearch using queries. The later sends the reports as email. PDF and CSV reports can be sent as attachments whiles HTML reports can be sent the email body.
Reports can be sent using event based triggers or time based triggers.
Creating a custom watch for a time based cron expects the following parameters:
a.
b.
c.
Elasticsearch versions supported by this plugin include :
| Version | - |
|---|---|
| Elassticsearch 5.4 | Testing (dev still in progress |
| Apache Kafka 0.11.0.0 | Testing (dev still in progress |
Measurement on how fast a trigger is sent to Apache Kafka after indexing and deleting data on Elasticsearch.
System Spec :
| Number of Events | Type | Trigger Active | Result |
|---|---|---|---|
| 1 | Indexed | Yes | - |
| 200 | 100 Indexed,100 Deleted | Yes | - |
| 2,000 | 1,000 Indexed, 1,000 Deleted | Yes | - |
| 2,000 | 1,000 Indexed, 1,000 Deleted | No | - |
| 2,000,000 | 1,000,000 Indexed, 1,000,000 Deleted | Yes | - |
| Elasticsearch Version | Comments |
|---|---|
| 5.4 | - |
Add the following configuration to your elasticsearch-kafka-watch.yml file located in /path/to/elasticsearch/config
Contributions are always welcome! Please read the contribution guidelines first.
Please read this.