/flowlog

Primary LanguageJavaApache License 2.0Apache-2.0

Description

Important: this project is only for demo purpose

The flowlog project is using flowlog-service, kafka, spark and cassandra to aggregate the realtime log. The aggregation has one-minute delay, which means the log storage is base on minute level. However, the apis only support hourly level query, it is not a hard limitation, we can easily change it to minute level log query.

For more detail, refer the Design Document, for the load test report, refer the ab testing result

How-to

  • Step 1, clone the repository and change directory to flowlog
  • Step 2, run command to initialize the cassandra, kafka, spark
gradle init_servers
  • Step 3, build and deploy the spark job
gradle submit_task
  • Step 4, launch 2nd terminal and start the service
gradle bootRun
  • Step 5, launch 3rd terminal and send sample requests
curl -kvvv -XPOST http://localhost:8080/flows -H'Content-Type: application/json'  -d '[{"hour":1,"src_app":"foo","desc_app":"bar","vpc_id":"vpc-1","bytes_tx":200,"bytes_rx":600},{"hour":1,"src_app":"foo","desc_app":"biz","vpc_id":"vpc-0","bytes_tx":1000,"bytes_rx":800}]]'
# we already prepare some sample data during init servers (refer Step 2)
curl -kvvv http://localhost:8080/flows\?hour\=1
curl -kvvv http://localhost:8080/flows\?hour\=2
# after 1 minute run
curl -kvvv http://localhost:8080/flows\?hour\=1
curl -kvvv http://localhost:8080/flows\?hour\=2
  • Step 6, stop the servers
gradle stop_servers
  • Step 7 (optional), restart the servers
gradle start_servers

Reference Documentation

For further reference, please consider the following sections: