/SeleniumAirflowDocker

Tuto about working with Selenium Airflow and Docker

Primary LanguageJupyter Notebook

Integrating Selenium, Airflow, Docker Tutorial

  • Type of challenge : Learning
  • Duration : 5 days

Mission objectives

  • Learning the basics of Airflow
  • Working with Selenium
  • Working with Docker
  • Ability to work with all three together

Version

Airflow version : 2.5.1 | Latest update: March 23, 2023.

Description

Why I am making this tutorial?

As a data engineer, I found myself needing to work with these three major technologies quickly, and I realized that most of the tutorials were outdated. So, I decided to create my own tutorial, as we all know that it can be challenging to learn solely by going through official documentation.

Structure of the repo

  • Main Branch: Here you will find the instructions to follow to complete the tutorial.
  • Corrected tuto: Here you'll find the completed tutorial.

If you get stuck at any point in the tutorial, feel free to switch to the 'Corrected_tuto' branch for assistance and to find more information on how these three technologies work together.

Let's start

  1. First of all, fork the repository so that you can work on it easily!
  2. Create your virtual environment
  3. Run 'pip install -r requirements.txt'.
  4. In the first folder, you will find the tutorial. Follow each step in numerical order.

Contributor

Cyril Verwimp, Junior Data Engineer

Connect with me: LinkedIn

"For the things we have to learn before we can do them, we learn by doing them." ― Aristotle