| title |
|---|
README |
This repo contains resources that summarize certain concepts in Spark or that help to perform certain operation with Spark. The scripts in this repos are made to make spark Debugging as easy as possible.
This piece of code is a simplistic helper to size your spark executors correctly based on certain limits. Its based on this wonderful article
Gives certain guidance on how to setup Ipython notebook with Spark and Yarn
When running Spark on yarn i faced many times spark executors dying while performing certain computations. The overall issue likely is Java 8 requesting too much virtual memory and in this folder im debugging this issue more thoroughly.