In23-S1-CS5229 - Big Data Analytics Technologies
- Year_wise_carrier_delay.py
- Year_wise_late_aircraft_delay.py
- Year_wise_NAS_delay.py
- Year_wise_security_delay.py
- Year_wise_Weather_delay.py
First, run the Year_wise_carrier_delay.py using the below command
spark-submit --master yarn Year_wise_carrier_delay.py
This is the command in which you can run the python script directly in the Hadoop terminal. Here Year_wise_carrier_delay.py contains all the steps I mentioned in running over the spark shell and it's a short way to run the query and find the query execution time as well.
Run other scripts as you want and see the execution logs to view the query execution time