/selenium-airflow

Using Airflow and Selenium to automate PM2.5 forecasting, from feature scraping (metereological data from earth.nullschool.net) to reporting.

Primary LanguagePython

Info

Using Airflow and Selenium to automate feature scraping (metereological data from earth.nullschool.net) and PM2.5 forecasting.

This repository is part of the 2nd project of 2110446 DATA SCIENCE AND DATA ENGINEERING. Group members and responsibilities (for this repository) are as follows:

  • Tarm Kalavantavanich: Airflow & Web scraping
  • Poom Suchao-in: Database
  • Phu Jongsaratis & Wipawan Temteerakit: Model

Installation

  1. run docker-compose up
  2. go to localhost:8080 (default)
  3. login (username: airflow, pw: airflow)
  4. run the DAG (needs internet connection!)