/streamingpro

Build Spark Streaming Application by SQL

Primary LanguageJavaScript

StreamingPro is a fast, expressive, and convenient system running on Spark with streaming, batch, interactive query, and machine learning support.

StreamingPro makes it easier to build spark applications without writing any code by means of:

  • Using json file in combination with modules, which are easy to be reused. This provides users declarative configurations to build spark applications.
  • Data processing is based on SQL.
  • Script support.

StreamingPro is not only an out-of-box complete application, but also an extensible and programmable framework for spark since you can develop you ower compositors(a.k.a moduler).

Features

  • Pure Spark Streaming (or normal Spark) program
  • No need of coding, only declarative workflows
  • Rest API for interactive querying
  • SQL-Oriented workflows support
  • Data continuously streamed in & processed in near real-time
  • dynamically CURD of workflows at runtime via Rest API
  • Flexible workflows (input, output, parsers, etc)
  • High performance
  • Scalable

Download

Download page: https://pan.baidu.com/s/1miDfTI4

streamingpro-spark-0.4.14-SNAPSHOT.jar for spark 1.6+, scala 2.10
streamingpro-spark-2.0-0.4.14-SNAPSHOT.jar for spark 2.0+, scala 2.11
streamingpro.flink-0.4.14-SNAPSHOT-online-1.2.0.jar for flink 1.2.0, scala 2.10

Documents

More Chinese articles: http://www.jianshu.com/c/759bc22b9e15

Architecture

If no picture is shown, please click me.

If github is too slow to view, please click me.

Declarative workflows for building Spark Streaming

If no picture is shown, please click me.

Implementation

If no picture is shown, please click me.