A sample workflow of using Docker Containers to run a Kafka producer and consumer to view Apple stock prices in real time.
- Docker Compose for simultaneously launching separate Docker Containers
- Apache Zookeeper and Apache Kafka
- Ruby-Kafka gem
- AlphaVantage API and AlphaVantageRB gem
- Timeloop gem to publish messages to the kafka topic at a set interval (at pace with when the updated streaming data is available)
navigate into the producer directory
$ cd producer
then install the Ruby dependencies
$ bundle install
repeat in the consumer directory
You will need to open 3 terminal windows. All commands should be issued from within the root project directory.
-
first window: run Docker Compose to set up the Zookeeper instance and sets up the Kafka cluster along with the "Apple" topic, 3 partitions and 1 replica for fault tolerance
$ docker-compose up
-
second window: launch the producer script to publish the data to the kafka message topic
$ ruby ./producer/src/producer.rb
-
third window: launch the consumer script to view the data from the kafka topic
$ ruby ./consumer/src/consumer.rb
$ CTRL + C
in each terminal window