Machine Learning with Apache Spark Notebooks
This project aims at teaching you the Apache Spark MLlib in python. It contains the example code and solutions to the exercises in O'Reilly upcoming book Machine Learning with Apache Spark by Adi Polak.
Quick Start
Want to play with these notebooks online without having to install anything?
TBD as we evaluate free platforms, if you have any recommendations, please reach out on @AdiPolak
At the moment, you can use docker with running the following command. Memory is for providing jupyter environment with more memory, --mount is for mounting the local library, where all you work will be saved. adipolak/ml-with-apache-spark is an image in docker hub. -p are the port used for interacting with jupyter notebook.
docker run -it --memory="28g" --memory-swap="30g" -p 8888:8888 --mount type=bind,source=$(pwd),target=/home/jovyan adipolak/ml-with-apache-spark
Just want to quickly look at some notebooks, without executing any code?
Brows it on the Juypyter.org notebook viewer
Want to run this project using a Docker image?
Read the Docker instructions - TBD
Want to install this project on your own machine?
TBD - would probably be using Docker.
FAQ
Which Python version should I use?
I get Py4JJavaError: An error occurred while calling o{some number}.parquet. (Reading Parquet file), what to do?
From Jupyter enter the terminal and validate your Java version, given that we use Spark 3.1.1 it should be openjdk version "11.0.11".
You local Java runtime should be of the same version - local Java runtime - is the Java that installed on the machine where you are running your docker from.
Which Apache Spark version should I use? The apache spark version is 3.1.1
Contributors
I want to thank everyone who contributed to this project by providing helpful feedback, filing issues, or submitting Pull Requests. If you would like to contribute, please feel free to submit a pull request, or reach out on @AdiPolak.