/koordinator-kafka-demo

Demo of the use of Koordinator to integrate batch and stream jobs

Primary LanguageJavaApache License 2.0Apache-2.0

CircleCI TypeScript

Koordinator Kafka Demo

This repository demos the use of Koordinator to integrate batch and stream based jobs. In this example, we have a batch job image-search, that given some search terms generate a list of urls of images matching the terms; and download-image, a Kafka producer/consumer written in Java that allows one to download the images in parallel.

Requirements

  • A Kafka broker, in our tests we use this docker container
  • Set the variables WORKER_USERNAME and WORKER_PASSWORD as your credentials in the Koordinator server, KOORDINATOR_URL as the url of your server and KAFKA_BROKER as the name of your Kafka broker.

Build

  1. Install the dependencies of the image-search worker: cd image-search && yarn install && cd ..
  2. Build the download-image worker: cd download-image && ./build.sh && cd ..

Run

Just run the script ./test.sh. It will start the workers and the workflow. You should be able to monitor its execution in the Koordinator frontend.

Submit a Pull Request

  1. Fork it!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request

License

Apache License V2