Download the required packages from requirements.txt file. API URL - https://rapidapi.com/vaccovidlive-vaccovidlive-default/api/vaccovid-coronavirus-vaccine-and-treatment-tracker/ Tools And Library Used::
Prefect - create the flow and dependencies between tasks. Project will be triggered from here.
PySpark - Written spark code in python.
DataBricks - Used as a platform for writing spark jobs.
Process Involved in execution of project:
- Create different notebook for different tasks in Databricks.
- Install Prefect
- Connect Prefect with Databricks
- Define the flow and dependencies between tasks in prefect
- Register and execute the task using prefect UI.