==========================
This project consists of several folders, each serving a specific purpose. Below is an overview of the folders and their contents:
-
conf: Contains the
install_python_dependencies.shfile used for bootstrapping in AWS before executing the PySpark script. -
src:
- spark-base: Implements individual models using scikit-learn libraries, focusing on parallel processing using PySpark.
- spark-mllib: Implements models using decision tree and random forest classifiers with Spark MLlib library.
-
input: Designated location for input files used by the PySpark scripts.
-
output: Stores the output of various runs from both the
spark-baseandspark-mllibprograms. -
logs: Contains AWS stdout/stderror logs for debugging or monitoring purposes.
-
Makefile: Provides commands and scripts to facilitate setup and execution of the project on an AWS bucket.
Please refer to the individual folders and their contents for more detailed information on each component of the project.