Market Ticks Streaming and Analytics With GridGain and Confluent Kafka

Confluent Kafka and GridGain/Ignite Market Ticks Streaming and Processing Demo.

The demo shows how to process market ticks using Confluent Kafka as a streaming engine and GridGain/Ignite as an in-memory database with persistence layer. Confluent and GridGain are hooked up together via GridGain Kafka Connect integration. SQL is a primary language used by Confluent and GridGain for data processing.

GridGain Cluster Setup and Launch

Launching the Demo Cluster

Download and decompress GridGain Enterprise Edition 8.8.19 or later
Move {gridgain}/libs/optional/control-center-agent folder to {gridgain}/libs
Navigate to {gridgain}/bin folder and start a server node(s): ./ignite.sh {demo_project_root}/src/main/resources/gridgain-cfg.xml
(Optional) Start several nodes

Monitoring the Demo Cluster

Sign up on portal.gridgain.com and go to Monitoring screen
Take the UUID from your server log and use it to add your cluster to Control Center
Now you have the cluster started and can monitor it!
Activate the cluster by going to the Cluster tab, select the '...' menu to the right of the cluster name and click 'Activate'

Create tables and seed data in GridGain

Start Bootstrapper class from your IDE to create demo tables and preload initial data

Confluent Platform and GridGain Setup

If Java 9+ is used by default then switch to Java 8 in a command line window to be used for Confluent scripts. Use this command on Mac OS:

export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/"

Install Confluent and GridGain Connect

Go to {gridgain}/integration/gridgain-kafka-connect/ and execute ./copy-dependencies.sh script from there
Download and extract Confluent Platform 7.7.1 or later
Go to {confluent}
Edit etc/kafka/connect-standalone.properties using your favourite text editor
Make the last line: plugin.path={gridgain}/integrations/gridgain-kafka-connect,{demo_project_root}/target/market-ticks-1.0-SNAPSHOT.jar
Save the file
Edit cfg/gridgain-sink.properties
Make sure igniteCfg points to your configuration file (the full path to src/main/resources/gridgain-cfg.xml)
Save

Prepare MarketOrder Source Connector

Build this project's JAR with Intellij IDEA or from your favourite IDE or using Maven from the project directory;

mvn clean package

Starting Confluent, Loading Source and Sink

Go to {confluent}
Start ZooKeeper: bin/zookeeper-server-start etc/kafka/zookeeper.properties
Start Kafka Server: bin/kafka-server-start etc/kafka/server.properties
Start the two Connectors: bin/connect-standalone etc/kafka/connect-standalone.properties {demo_project_root}/cfg/gridgain-sink.properties {demo_project_root}/cfg/market-orders-source.properties
(Optional) Start the Kafka Control Center: bin/control-center-start etc/confluent-control-center/control-center-minimal.properties. You will be able to access it at http://localhost:9021/

Real-Time Analytics with GridGain

Events will be generated and arriving in the cluster in real-time. Run sample queries like those below for the demo purpose:

Total spending per buyer;

SELECT first_name, last_name, SUM(bid_price) as mp
FROM MarketOrder as m JOIN Buyer as b ON m.buyer_id = b.id
GROUP BY first_name, last_name ORDER BY mp DESC;

Cash Distribution (fraction of money spent per symbol);

SELECT symbol, SUM(bid_price) as mp
FROM MarketOrder GROUP BY symbol ORDER BY mp DESC;

Restart GridGain cluster (activate with Control Center or the CMD tool if needed) and show that the day is still in the cluster.

GridGain-Demos/ignite-kafka-market-ticks-demo