theho/awesome-spark

A curated list of awesome Apache Spark packages and resources.

Awesome Spark

---

A curated list of awesome Apache Spark packages and resources.

• Packages

  • Language bindings
  • Notebooks and IDEs
  • General purpose libraries
  • SQL Data Sources
  • Bioinformatics
  • GIS
  • Time series analytics
  • Graph processing
  • Machine Learning Extension

• Resources

  • Books
  • MOOCS
  • Workshops
  • Projects Using Spark

Language bindings

• Flambo - Clojure DSL.
• Mobius - C# bindings.

Notebooks and IDEs

• Apache Zeppelin
• Spark Notebook

General purpose libraries

• Succinct - Support for efficient queries on compressed data.

SQL Data Sources

• Spark CSV - CSV reader and writer.
• Spark Avro - Apache Avro reader and writer.
• Spark XML - XML parser and writer.
• Spark-Mongodb - MongoDB reader and writer.
• Spark Cassandra Connector - Cassandra support including data source and API and support for arbitrary queries.
• Spark Riak Connector - Riak TS & Riak KV connector
• Mongo-Spark - Official MongoDB Spark Connector

Bioinformatics

• ADAM - a set of tools designed to analyse genomics data.

GIS

• Magellan - Geospatial Analytics Using Spark.
• GeoSpark - A Cluster Computing System for Processing. Large-Scale Spatial Data

Time series analytics

• Spark-Timeseries - A Scala / Java / Python library for interacting with time series data on Apache Spark.

Graph processing

• Mazerunner - graph analytics platform on top of Neo4j and GraphX.
• GraphFrames - DataFrame-based graph API.

Machine Learning Extension

• dbscan-on-spark - An Implementation of the DBSCAN clustering algorithm on top of Apache Spark by irvingc and based on the paper from He, Yaobin, et al. MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data.
• spark_dbscan - Another Implementation of the DBSCAN clustering algorithm on top of Apache Spark by alitouka.
• Apache SystemML - declarative machine learning framework on top of Spark.
• Mahout Spark Bindings - linear algebra DSL and optimizer with R-like syntax.
• spark-sklearn - scikit-learn integration with distributed model training.
• KeystoneML - type safe machine learning pipelines with RDDs.

Books

• Mastering Apache Spark.
• Learning Spark, Lightning-Fast Big Data Analysis.
• Advanced Analytics with Spark.

MOOCS

• CS100 - Introduction to Big Data with Apache Spark.
• CS190 - Scalable Machine Learning.

Workshops

• AMP Camp

Projects Using Spark

• Oryx 2 - a lambda architecture built on Apache Spark and Apache Kafka with specialization for real-time large scale machine learning.
• PredictionIO - Machine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time.

---

This work (Awesome Spark, by https://github.com/awesome-spark/awesome-spark), identified by Maciej Szymkiewicz, is free of known copyright restrictions.