This project is a quick example of how to get data out of kafka in realtime, do some stream processing to make it more effective in charts, and then PUSH it to a web client, rather than the old paradigm of pull and batch.
This example leverages Websockets (Socket.io NPM), however I have also implemented this in Server Sent Events, which I will probably integrate into this repo so you can run either/both depending on your requirements.
This assumes you've run Confluent tarball (or have another Confluent Kafka environment running). It's assumes non-secured cluster and thus a non-SASL client config, in the client config of kafkajs-vis-server.js
you'll see a commented section for if you do need to specify jaas config.
https://docs.confluent.io/current/quickstart/ce-quickstart.html
Make sure you have Node.js and NPM installed!
Install all NPM prereqs by running the following command while inside the cloned repo folder:
npm install
NOTE: You shouldn't need to get anything extra for ChartJS Streaming Plugin; However, if you want to update it to latest, get it from here and put in the dist folder: https://github.com/nagix/chartjs-plugin-streaming
- Fire up a terminal and go into your confluent home directory (in this case 5.4.0 tarball).
cd ~/Confluent/confluent-5.4.0
..then create our topics we'll use with the following:
bin/kafka-topics --create --topic TRUCK_ENGINE_SENSORS --replication-factor 1 --partitions 1 --bootstrap-server localhost:9092
bin/kafka-topics --create --topic TRUCK_1_SENSORS --replication-factor 1 --partitions 1 --bootstrap-server localhost:9092
bin/kafka-topics --create --topic TRUCK_2_SENSORS --replication-factor 1 --partitions 1 --bootstrap-server localhost:9092
bin/kafka-topics --create --topic TRUCK_3_SENSORS --replication-factor 1 --partitions 1 --bootstrap-server localhost:9092
..then run consumer to watch the records come through a command line:
bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic TRUCK_ENGINE_SENSORS --from-beginning
- Fire up another terminal and go into the following directory (assuming you cloned this into your home directory):
cd ~/Kafka_KSQL_PushVis/truck_sensors/producers
..and run the producer script..
./producers.sh
(take a look at that shell script, tweak as desired, current parameters work well)
- In a fresh console run
ksql
..then, open the truck_streams.ksql
file in a text editor and walk through each section via the KSQL CLI, copy/paste into ksql and run, ensure each is successful.
- Run a select query on the ingest topic and produce data, see it coming through
Select * from TRUCK_ENGINE_SENSORS;
Then show that TRUCK_1_ENGINE_SENSORS is getting data to it due to the filtering/output stream we did in one of the ksql create statements.
Select * from TRUCK_1_ENGINE_SENSORS;
-
We're now ready to let'r rip..
node kafkajs-vis-server.js
-
Open the web page (index.html) and watch the data roll in
Note also for showing how bidirectinal websocket communication works
- Push the start/stop producing buttons and look at correspending code in kafkajs-vis-server.js
- Click into the input field under the vis run a command like
ls
and hit enter.. show that we can execute commands on the remote system (that's how start/stop producer runs).
No problem, go into kafkajs-vis-server.js
and edit the bootstrap server url in the brokers
parameter. If you need ssl and/or sasl (aka Confluent Cloud creds), uncomment that section under the brokers
parameter and you're good to go!
With this you can now go create a modern event-first visualization platform. Go forth and profit!