- Illustrate a small process of data engineering
- Creation of fake data: stores that have several sensors to count visitors and send data hourly
- API creation and API requests for Data extraction
- Data transformation: creation of new stats (daily traffic, moving average for each weekday)
- WebApp creation for data visualisation
- Using workflows to check code syntax (black for PEP8)
- Create a new virtual environment, using poetry, venv, conda
- run
pip install -r requirements.txt
- The first part of this project is to create fake data
- It should be requestable with an API
- Fake data creation using numpy
- Unit tests for Sensor and Store classes
python tests/test_sensors.pypython tests/test_store.py
- Creation of an api with FastAPI
- We create it to simulate the provider’s API, here the API is deployed locally.
- To launch the api locally, run
uvicorn app:app --reload
The goal is to request the API to build our data. You must deploy the API locally before running the script.
- Computation of the daily traffic by store
- Computation of the moving average daily traffic for the same day of the week over the last 4 weeks
- Computation of this moving average change from one week to the next
- Export to parquet file
- Creation of a streamlit webapp
- Choice of a store and a sensor to display its data and barplots about its most recent stats.
- Run the app using
streamlit run app_streamlit.py
- Add alert if the value of a sensor is below a fixed threshold
- Containerize the repo in a Docker container to run it on the cloud
- Store the data on the cloud