Limbo

Raison d'être:

Note: Limbo is in a WIP state.

Limbo is a Scala API which allows to leverage best of different data processing frameworks by allowing seamless transition between framework specific data structures.

Features

Itegration between Scio and Spark
Programmatic Spark job submission to a Apache YARN cluster
Scala API for Google Dataproc cluster

Example:

// Start in Scio:
val (sc, args) = ContextAndArgs(argv)
val scol = sc.parallelize(1 to 10)

// Move to Spark realm
scol.toRDD().map { rdd =>
  rdd
    .map(_ * 2)
    .saveAsTextFile(args("output"))
}

Code of conduct

This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

License