A high-level framework for organizing and submitting PySpark jobs via spark-submit
.
Run Makefile
This will create a new folder dist
, which will house .zip files of our jobs
module as well as local .zip files of all the dependency packages.
make build
Cd into dist
and run a Spark Submit command.
cd dist && spark-submit --py-files jobs.zip main.py --job wordcount
-
src
: Holds Python code and copies of requried packages.--
jobs
: Individual PySpark jobs, which can be specified at the command line withspark-submit
.--
libs
: Dependency libraries that are downloaded from pip at or before runtime.--
shared
: Code shared by all jobs.--
main.py
: The main Python script that defines command line arguments and SparkContext. -
tests
: Unit testing for all jobs--
_jobs
: Directory containing sub-directories for each job and test files. -
requirements.txt
: A list of third-party packages that will be automatically installed from pip into thesrc/libs
directory at runtime. -
dev_requirements.txt
: Dependency packages for thepyspark
package and others. -
Makefile
: Installs dependencies and runs tests.