Using this awesome project to expand and hopefully build a workable big data streaming example for learning Kafka, Spark, SuperSet, and Machine Learning.
Integrating Faker generated data to create user accounts to withdraw and deposit random amounts dependent on a user classification of their spend type: ** Spend Types: LOW, MEDIUM, HIGH
This is the supporting repository for my blog post: Building A Streaming Fraud Detection System With Kafka And Python.
This fraud detection system is fully containerised. You will need Docker and Docker Compose to run it.
You simply need to create a Docker network called kafka-network
to enable communication between the Kafka cluster and the apps:
$ docker network create kafka-network
All set!
- Spin up the local single-node Kafka cluster (will run in the background):
$ docker-compose -f docker-compose.kafka.yml up -d
- Check the cluster is up and running (wait for "started" to show up):
$ docker-compose -f docker-compose.kafka.yml logs -f broker | grep "started"
- Start the transaction generator and the fraud detector (will run in the background):
$ docker-compose up -d
Show a stream of transactions in the topic T
(optionally add --from-beginning
):
$ docker-compose -f docker-compose.kafka.yml exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic T
Topics:
queuing.transactions
: raw generated transactionsstreaming.transactions.legit
: legit transactionsstreaming.transactions.fraud
: suspicious transactions
Example transaction message:
{"source": "yGfZ1Xa6k1r0", "target": "N5RvY7RO5sQF", "amount": 217.46, "currency": "EUR"}
To stop the transaction generator and fraud detector:
$ docker-compose down
To stop the Kafka cluster (use down
instead to also remove contents of the topics):
$ docker-compose -f docker-compose.kafka.yml stop
To remove the Docker network:
$ docker network rm kafka-network