Scio

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]

Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Features

Scala API close to that of Spark and Scalding core APIs
Unified batch and streaming programming model
Fully managed service^*
Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable
HDFS, JDBC, TensorFlow TFRecords, Cassandra, Elasticsearch and Parquet I/O
Interactive mode with Scio REPL
Type safe BigQuery
Integration with Algebird and Breeze
Pipeline orchestration with Scala Futures
Distributed cache

^* provided by Google Cloud Dataflow

Quick Start

The ubiquitous word count example can be run directly with SBT in local mode, using README.md as input.

sbt "scio-examples/run-main com.spotify.scio.examples.WordCount --input=README.md --output=wc"
cat wc/part-00000-of-00001.txt

Documentation

Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.

Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.

Scio Wiki - wiki page
Scio Scaladocs - current API documentation

Artifacts

Scio includes the following artifacts:

scio-core: core library
scio-test: test utilities, add to your project as a "test" dependency
scio-avro: add-on for Avro, included in scio-core but can also be used standalone
scio-bigquery: add-on for BigQuery, included in scio-core but can also be used standalone
scio-bigtable: add-on for Bigtable
scio-cassandra2: add-on for Cassandra 2.x
scio-cassandra3: add-on for Cassandra 3.x
scio-elasticsearch2: add-on for Elasticsearch 2.x
scio-elasticsearch5: add-on for Elasticsearch 5.x
scio-extra: extra utilities for working with collections, Breeze, etc.
scio-hdfs: add-on for HDFS IO
scio-jdbc: add-on for JDBC IO
scio-parquet: add-on for Parquet
scio-tensorflow: add-on for TensorFlow TFRecords IO and prediction

License