/airflow-docker-gpu

Primary LanguagePython

setup

start postgres

docker-compose up -d postgres_airflow

init db and add user

docker-compose run airflow_init

run webserver

docker-compose up -d airflow_webserver

run scheduler

docker-compose up airflow_scheduler

running

  • web server is exposed on port 9090
  • login admin/airflow
  • dag_id is gpu_docker_dag
  • task logs can be found in ./logs

reproduce

  • uncomment line in docker-compose
  • update docker-py in the airflow_scheduler service:
    • pip install --upgrade docker>=4.3.0
  • run dag gpu_docker_dag
  • task no_gpu fails, with_gpu succeeds