makism/spark-template

A general pyspark template (defunkt).

ETL

You could run some of the parts like

docker exec -it spark-master /bin/python3 /opt/etl/template/raw/compute.py

Tests

docker exec -it spark-master /usr/local/bin/pytest -v /opt/etl_tests