/Arrow-Task-Engine

This repository provides an example task engine utilising Apache Arrow.

Primary LanguagePython

Arrow Task Engine

This example tutorial shows you how to build a simple task engine using Arrow, docker and an analytics library called "Anomaly Detection Tool Kit".

Architecture

"architecture"

Setup

Please make sure you have the following prerequisites before you begin:

InfluxDB IOx

To get started, you will need to create an InfluxDB IOx account. If you don't already have an account, you can sign up for free here. Once you have an account, you can create a new organization and bucket to store your data. You can find instructions on how to do this here.

Env file

To connect Grafana to InfluxDB IOx, you will need to create an env file. With the top directory create a file called .env:

touch .env

This file will contain the following information:

export INFLUX_HOST=
export INFLUX_TOKEN=
export INFLUX_ORG=
export INFLUX_BUCKET=

Note: make sure to specify your INFLUX_HOST without the protocal like so: us-east-1-1.aws.cloud2.influxdata.com

Grafana flighsql plugin

To connect Grafana to InfluxDB IOx, you will need to install the Grafana flighsql plugin. To do this follow these instructions here.

Once you have downloaded the plugin, unzip the folder and copy the influxdata-flightsql-datasource directory to the plugins directory located in the grafana directory.

Run

To run, make sure to first source the env file:

source .env

Navigate to the quick start you would like to run and run the following command:

docker-compose up -d

This will start Grafana server and Telegraf. Once the quick start is running, you can access Grafana at localhost:3000. You can log in with the default username and password: admin and admin.