This is the git repo for learning airflow pipeline implementation for machine learning use case. The code has been improvised from the source repositories:
- e2e-ml-pipeline-airflow - https://github.com/NicoloAlbanese/airflow-ml-pipeline-mvp
- pycon-sweden-airflow-ml-pipelines - https://github.com/pycon-ml/airflow_workshop
You should have docker
and docker-compose
installed on your machine !
The easiest way to have everything ready for the workshop is to install Docker Desktop
Minimum resource requirement for docker to start all the services is mentioned below:
Resource | Recommendation |
---|---|
Memory | 3 GB |
CPU | 2 CPU |
Clone the repo and cd into corresponding folder.
docker-compose up
Username: airflow
Password: airflow
Stop and remove containers, networks, images, and volumes
docker-compose down