This provides a template to work with Apache Spark and with Ignition-Core.
- This template will point to a stable Spark and Hadoop-client version, potentially working-around some known bugs
src/main/scala/ignition/jobs/Runner.scala
is a example of multi-setup project that can run usingcore/tools/cluster.py jobs run
command- See
scripts/job_runner.py
for a full-featured, production-grade script-template that launchs and supervises clusters and jobs using AWS spot instances with automatic failure recovery
In general, check the source files for more information.
See http://monkeys.chaordic.com.br/start-using-spark-with-ignition/ for quick-start tutorial
- An example using Scalaz Validation
- Other examples, more documentation