CDC (Change Data Capture)
Change Data Capture platform example using MySQL, Maxwell, Kafka, Storm and Elasticsearch.
Environment Setup
I assume you have locally installed storm, docker-machine, docker-compose and MySQL running on port 3306.
docker-machine create --driver virtualbox --virtualbox-memory 4096 --virtualbox-cpu-count 2 cluster-cdc
MYSQL_HOST=10.0.2.2 MYSQL_PORT=3306 MYSQL_USER=root MYSQL_PASSWORD=root ADVERTISED_HOST=$(docker-machine ip cluster-cdc) docker-compose up
Build Storm topology
To manage java versions I use jenv.
jenv local 1.6
jenv exec mvn package -DskipTests=true
The example topology works in a pass through mode, where you can easily plug in and experiment with custom analysis and data format manipulation.
Run Storm topology
jenv exec storm jar target/cdc-topology-1.0.0-jar-with-dependencies.jar cdc.storm.KafkaEsTopology KafkaEs -c nimbus.host=$(docker-machine ip cluster-cdc) -c nimbus.thrift.port=49627
Exposed ports
Storm UI
open http://$(docker-machine ip cluster-cdc):49080
Kibana
open http://$(docker-machine ip cluster-cdc):5601
Cleanup
docker-machine stop cluster-cdc