/scio

A Scala API for Google Cloud Dataflow and Apache Beam

Primary LanguageScalaApache License 2.0Apache-2.0

Scio

Build Status codecov.io GitHub license Maven Central Join the chat at https://gitter.im/spotify/scio

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]

Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Google Cloud Dataflow and Apache Beam inspired by Spark and Scalding. See the current API documentation for more information.

Scio is being donated to Apache Beam as a Scala DSL (BEAM-302).

Features

  • Scala API close to that of Spark and Scalding core APIs
  • Unified batch and streaming programming model1, 2
  • Fully managed service2
  • Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable2
  • HDFS source/sink
  • Interactive mode with Scio REPL
  • Type safe BigQuery
  • Integration with Algebird and Breeze
  • Pipeline orchestration with Scala Futures
  • Distributed cache

1 provided by Apache Beam

2 provided by Google Cloud Dataflow

Quick Start

The ubiquitous word count example can be run directly with SBT in local mode, using README.md as input.

sbt "scio-examples/run-main com.spotify.scio.examples.WordCount --input=README.md --output=wc"
cat wc/part-00000-of-00001.txt

Documentation

Artifacts

Scio includes the following artifacts:

  • scio-core: core library
  • scio-test: test utilities, add to your project as a "test" dependency
  • scio-bigquery: Add-on for BigQuery, included in scio-core but can also be used standalone
  • scio-bigtable: Add-on for Bigtable
  • scio-extra: Extra utilities for working with collections, Breeze, etc.
  • scio-hdfs: Add-on for HDFS

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0