/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree

⌚ 🧳 ✈️ A project based in High Performance Computing. This project was built using Apache Spark API, Java and Gradle. The scenario of the project was a Cluster-based implementation of the Prim's Algorithm in a Graph representation of a network of routes between several airports and the average departure delays of that routes. The goal of this project is to find the bottleneck airport, i.e., the airport with greater aggregated departure delays, then apply a reduction factor and recompute the Minimum Spanning Tree (M.S.T. - Prim's Algorithm).

Primary LanguagePythonMIT LicenseMIT

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm)

https://raw.githubusercontent.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree/master/imgs/JPGs/banner-1.jpg

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Banner #1

Current Status

contributor for this repository developed in current version

status of this version no. 1 status of this version no. 2 status of this version no. 3 status of this version no. 4

keyword of this version no. 1 keyword of this version no. 2 keyword of this version no. 3 keyword of this version no. 4

technology used no. 1 technology used no. 2 technology used no. 3 software used no. 1 software used no. 2

star this repository fork this repository downloads of this repository price of this project

Current Progress of the Project

current progress of this project

Final Approval Grade of the Project

grade of this project

Description

A 1st year's lab work (project) of the MSc. degree of Computer Science and Engineering made in FCT NOVA (Faculty of Sciences and Technology of New University of Lisbon), in the subject of High Performance Computing. This project was built using Apache Spark API, Java and Gradle. The scenario of the project was a Cluster-based implementation of the Minimum Spanning Tree (Prim's algorithm), with the support of Apache Spark's API, using its offered cluster-based parallel optimisations and operations.

Click here to see the Project's Description!

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Project's Description #1

Screenshots

https://raw.githubusercontent.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree/master/imgs/JPGs/screenshot-1.jpg

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Screenshot #1

https://raw.githubusercontent.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree/master/imgs/JPGs/screenshot-2.jpg

Apache Spark API - Flight Airports’ Average Departure Delays Minimum Spanning Tree (Prim’s Algorithm) - Screenshot #2

Getting Started

Prerequisites

To install and run this application, you will need:

The Git feature and/or a third-party Git Client based GUI, like:

Installation

To install this application, you will only need to download or clone this repository and run the application locally:

You can do it downloading the .zip file in download section of this repository.

Or instead, by cloning this repository by a Git Client based GUI, using HTTPS or SSH, by one of the following link:

https://github.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

git@github.com:rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

Or even, by running one of the following commands in a Git Bash Console:

git clone https://github.com/rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

git clone git@github.com:rubenandrebarreiro/apache-spark-api-flight-airports-average-departure-delays-minimum-spanning-tree.git

Built with

Notes

  • There's some useful parameters which you can try in as JVM (Java Virtual Machine) parameters, for a better Memory and Java Heap Space management, in the case of be running the application locally, instead of running the application in a cluster system:
Option #1
-Xms8g
-Xmx16g
-XX:MinHeapFreeRatio=50
-XX:MaxHeapFreeRatio=50
-XX:NewRatio=3
-XX:NewSize=2048m
-XX:MaxNewSize=2048m
-XX:+AggressiveHeap
Option #2
-Xms12g
-Xmx16g
-XX:MinHeapFreeRatio=50
-XX:MaxHeapFreeRatio=50
-XX:NewRatio=3
-XX:NewSize=2g
-XX:MaxNewSize=2g
-XX:+AggressiveHeap

Contributors

Rúben André Barreiro

Contacts

Rúben André Barreiro

E-mails

Portfolios/Blogs and Git Hosting/Repository Services

Rúben André Barreiro

GitHub's Portfolio/Personal Blog

Hosting/Repository Services

Classifications/Final Approval Grades

Approval Grade of Lab Work

  • 18 of 20