cmusjtuliuyuan/BigDL-Tutorials

Step-by-step Deep Leaning Tutorials on Apache Spark using BigDL

Deep Leaning Tutorials on Apache Spark using BigDL

Step-by-step Deep Leaning Tutorials on Apache Spark using BigDL. The tutorials are inspired by Apache Spark examples, the Theano Tutorials and the Tensorflow tutorials.

Topics

1. RDD
2. DataFrame
3. SparkSQL
4. StructureStreaming
5. Forward and backward
6. Linear Regression
7. Introduction to MNIST
8. Logistic Regression
9. Feedforward Neural Network
10. Convolutional Neural Network
11. Recurrent Neural Network
12. LSTM
13. Bi-directional RNN
14. Auto-encoder

Environment

• Mac OS / Linux
• Python 2.7 (Required python libraries: numpy, scipy, pandas, scikit-learn, matplotlib. You may want to use pip to install these packages.)
• Apache Spark 2.1.0
• Jupyter Notebook 4.1
• BigDL 0.1.1(download linux64, mac)

Start Jupyter Server

• Create start_notebook.sh, copy and paste the contents below, and edit SPARK_HOME, BigDL_HOME accordingly.

#!/bin/bash

#setup pathes
SPARK_HOME= where the downloaded spark
BigDL_HOME= where the downloaded file unzipped

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --notebook-dir=./ --ip=* --no-browser"

source ${BigDL_HOME}/bin/bigdl.sh
${SPARK_HOME}/bin/pyspark \
  --master local[4] \
  --driver-memory 4g \
  --properties-file ${BigDL_HOME}/conf/spark-bigdl.conf \
  --py-files ${BigDL_HOME}/lib/bigdl-0.1.1-python-api.zip \
  --jars ${BigDL_HOME}/lib/bigdl-SPARK_2.1-0.1.1-jar-with-dependencies.jar \
  --conf spark.driver.extraClassPath=${BigDL_HOME}/lib/bigdl-SPARK_2.1-0.1.1-jar-with-dependencies.jar \
  --conf spark.executor.extraClassPath=${BigDL_HOME}/lib/bigdl-SPARK_2.1-0.1.1-jar-with-dependencies.jar

• Execute start_notebook.sh in bash, it will start a jupyter notebook service and output the url to access

Run Demo

• Open a browser - Suggest Chrome or Firefox or Safari
• Access notebook client at address http://localhost:8888/?token=xxxx (which is in the output of the start_notebook.sh), open the example ipynb files and execute.