simulates sending e-store purchase data generated using Faker and a Simon model
- Data is generated using the
produce
entrypoint which sends data to Kafka - A consumer picks that data up from the
purchases
topic in Kafka - The consumer ensures the data is in a clean format and pushes to postgres
- Stats are exposed on the data via the ibis-framework library in a dashboard in Dash (in progress)
- Practice tools I am not as proficient as I would like to be such as setting up databases/containers/apps
- Show off
- [] Add consumer
- [] Functionality for consumer of cleaning data and pushing to postgres
- [] Retention policies to Kafka as to not store data forever
- [] Expose ibis and a simple Dash app against the postgres database
- [] Make the producer more continuous with a cron