/data-algorithms-book

MapReduce and Spark Source Code and Scripts for Data Algorithms Book

Primary LanguageJavaOtherNOASSERTION

Data Algorithms: Recipes for Scaling up with Hadoop and Spark

Data Algorithms Book

Author book signings for ("Data Algorithms") will be held in the O'Reilly booth on Thursday, Feb. 19, 2015. Complimentary copies of books will be provided for the first 25 attendees.

I have started adding bonus chapters.

Repository

This repository will host all source code and scripts for Data Algorithms Book. This book provides a set of distributed MapReduce algrithms, which are implemented using

  • Java (JDK7)
  • Spark 1.2.0
  • MapReduce/Hadoop 2.6.0

Work in Progress...

Please note that this is a work in progress...

Data Algorithms Book Work In Progress

URL To Data Algorithms Book

Source Code

  • All source code, libraries, and build scripts are posted here
  • Shell scripts are posted for running Spark and Mapreduce/Hadoop programs (in progress...)

Software Used

Software Version
Java JDK7
Hadoop 2.6.0
Spark 1.2.0
Ant 1.9.4

Structure of Repository

Name Description
README.md The file you are reading now
README_lib.md Must read before you build with Ant
src Source files for MapReduce/Hadoop/Spark
scripts Shell scripts to run MapReduce/Hadoop and Spark pograms
lib Required jar files for compiling source code
build.xml The ant build script
dist The ant build's output directory (creates a single JAR file)
LICENSE License for using this repository (Apache License, Version 2.0)
misc misc. files for this repository
setenv example of how to set your environment variables before building
data sample data files (such as FASTQ and FASTA) for basic testing purposes

Structure of src Directory

src directory

Also, each chapter has two sub folders:

org.dataalgorithms.chapNN.spark      (for Spark programs)
org.dataalgorithms.chapNN.mapreduce  (for Mapreduce/Hadoop programs)

where NN = 00, 01, ..., 31

How To Build using Apache's Ant

How To Build by Ant

Sample Builds by Ant

How To Run Spark/Hadoop Programs

How To Run Python Programs

To run python programs just call them with spark-submit together with the arguments to the program.

Questions/Comments

Thank you!

best regards,
Mahmoud Parsian