This project implements data streaming techniques to detect anomalies on streams of data. Given a series of events the system can detect heavy hitters and heavy changers: the former are those events whose frequency of occurrence is higher than a certain threshold value on a given time windows, and the latter are those events showing a difference on their frequency of occurrence on two adjacent time windows that is greater than another (possibly equal) threshold value.
There are a few changes you may want to make for configuring the service parameters. Relevant configuration sections are listed below. Configuration file: sketch-ws/config.yml.
sketchConfig
: this parameters define the shape of the sketches created bySketchManager
on each detection epoch. Sub-parameterprime
is used to do some modulo arithmetic and, as it name implies, must be a prime number. Keep in mind thatprime>rows
, otherwise the service will fail to start.detectionParameters
: here you can change heavy hitters and heavy changers thresholds. ParametersketchRotationInterval
define the time window length (in seconds). Finally, the number of sketches stored bySketchHistoryQueue
and heavy keys stored byHeavyKeysHistoryQueue
can be adjusted with sub-parametermaxHistoryQueueLength
.
The system can by started by using the docker-compose
file located on sketch-ws/ci/compose/:
The compose file will build the service Dockerfile image located at sketch-ws/ci/Dockerfile-receiver/. This build process uses two files that must be generated by using the script sketch-ws/ci/Dockerfile-receiver/copy_dockerfile_files.sh (a docker image will be automatically generated with a new push so don't forget to include the .yml
and .jar
files).
$ cd ci/Dockerfile-receiver/
$ ./copy_dockerfile_files.sh
$ cd ../compose
$ docker-compose up
You will see how the service spins up. Available resources will be shown on the console:
GET /health (com.lpineda.dsketch.resources.Health)
GET /heavykeys (com.lpineda.dsketch.core.HeavyKeysHistoryQueue)
GET /sketchhistory (com.lpineda.dsketch.core.SketchHistoryQueue)
GET /status (com.lpineda.dsketch.resources.Status)
More at: