/awesome-streaming

a curated list of awesome streaming frameworks, applications, etc

Awesome Streaming Awesome Build Status

A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources. Inspired by other awesome projects.

Table of Contents

Streaming Engine

  • Apache Apex [Java] - unified platform for big data stream and batch processing.
  • Apache Flink Streaming [Java] - system for high-throughput, low-latency data stream processing that supports stateful computation, data-driven windowing semantics and iterative stream processing.
  • Apache Gearpump [Scala] - lightweight real-time distributed streaming engine built on Akka.
  • Apache Ignite Streaming [Java] - Ignite streaming allows to process continuous never-ending streams of data in scalable and fault-tolerant fashion.
  • Apache Samza [Scala/Java] - distributed stream processing framework that build on Kafka(messaging, storage) and YARN(fault tolerance, processor isolation, security and resource management).
  • Apache Spark Streaming [Scala] - makes it easy to build scalable fault-tolerant streaming applications.
  • Apache Storm [Clojure/Java] - distributed real-time computation system. Storm is to stream processing what Hadoop is to batch processing.
  • Hazelcast Jet [Java] - A general purpose distributed data processing engine, built on top of Hazelcast.
  • IBM Streams [Python/Java/Scala] - platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)
  • heron - Twitter's real-time analytics platform that is fully API-compatible with Storm. Storm has been replaced by Heron at Twitter.
  • mantis - Netflix's event stream processing system.
  • millwheel - framework for building low-latency data-processing applications that is widely used at Google.
  • mupd8(muppet) [Scala/Java] - mapReduce-style framework for processing fast/streaming data.
  • s4 [Java] - general-purpose, distributed, scalable, fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.
  • SPQR [Java] - dynamic framework for processing high volumn data streams through pipelines.
  • tigon [C++/Java] - high throughput real-time streaming processing framework built on Hadoop and HBase.
  • hailstorm [Haskell] - distributed stream processing with exactly-once semantics based on Storm.
  • Teknek [Java] - Simple elegant stream processing with interactive prototying shell SOL (Stream Operator Language)
  • concord [C++] - a distributed stream processing framework built in C++ on top of Apache Mesos, designed for high performance data processing jobs that require flexibility & control.
  • StreamBox - a modern stream processing engine which can handle out-of-order streaming data and achieve high throughput with low latency.
  • Wallaroo [Python] - The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.

Streaming Library

  • IBM Streams [Python/Java/Scala] - platform for distributed processing and real-time analytics. Provides toolkits for advanced analytics like geospatial, time series, etc. out of the box.
  • Apache Kafka Streams [Java] - lightweight stream processing library included in Apache Kafka (since 0.10 version).
  • Akka Streams [Scala] - stream processing library on Akka Actors.
  • Swave [Scala] - A lightweight Reactive Streams Infrastructure Toolkit for Scala.
  • FS2(prev. 'Scalaz-Stream') [Scala] - Compositional, streaming I/O library for Scala.
  • monix [Scala] - high-performance Scala / Scala.js library for composing asynchronous and event-based programs.

Streaming Application

  • straw [Python/Java] - A platform for real-time streaming search.
  • storm-crawler [Java] - Web crawler SDK based on Apache Storm.

IoT

  • sensorbee [Go] - lightweight stream processing engine for IoT.
  • Apache Edgent [Java] - a programming model and runtime that enables continuous streaming analytics on gateways and edge devices which can work with centralized systems to provide efficient and timely analytics across the whole IoT ecosystem: from the center to the edge, opens sourced by IBM.

DSL

  • summingbird [Scala] - library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including Storm and Scalding.
  • coast [Scala] - a DSL that builds DAGs on top of Samza and provides exactly-once semantics.
  • Apache Beam [Java] - unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs), open sourced by Google.
  • Esper [Java] - component for complex event processing (CEP) and event series analysis.
  • Streamparse [Python] - lets you run Python code against real-time streams of data via Apache Storm.

Data Pipeline

  • Apache Kafka [Scala/Java] - distributed, partitioned, replicated commit log service, which provides the functionality of a messaging system, but with a unique design.
  • Apache Pulsar [Java] - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
  • metaq [Java] - Taobao's high available, high performance distributed messaging system
  • nsq [Go] - realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.
  • NATS streaming [Go] - fast disk-backed messaging solution
  • camus [Java] - Linkedin's Kafka -> HDFS pipeline.
  • databus [Java] - Linkedin's source-agnostic distributed change data capture system.
  • flume [Java] - distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
  • Redis Streams - Redis implementation of a unified log structure
  • suro [Java] - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data.
  • StreamSets Data Collector [Java] - continuous big data ingestion infrastructure that reads from and writes to a large number of end-points, including S3, JDBC, Hadoop, Kafka, Cassandra and many others.

Online Machine Learning

  • streamDM [Scala] - mining Big Data streams using Spark Streaming from Huawei.
  • jubatus [C++] - distributed processing framework and streaming machine learning library.
  • Apache Samoa [Java] - distributed streaming machine learning (ML) framework that contains a programing abstraction for distributed streaming ML algorithms.
  • trident-ml [Java] - realtime online machine learning library based on Trident.
  • StormCV [Java] - enables the use of Apache Storm for video processing by adding computer vision (CV) specific operations and data model.
  • DataSketches [Java] - sketches library from Yahoo!.
  • StreamingBandit [Python] - Provides a webserver to quickly setup and evaluate possible solutions to contextual multi-armed bandit (cMAB) problems.

Streaming SQL

  • pipelinedb [C] - An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
  • squall [Java] - Squall executes SQL queries on top of Storm for doing online processing.
  • StreamCQL [Java] - Continuous Query Language on RealTime Computation System.
  • KSQL [Java] - a Streaming SQL Engine for Apache Kafka.

Benchmark

  • storm-benchmark [Java] - a set of benchmarks to test Storm performance.
  • storm-perf-test [Java] - a simple storm performance/stress test.
  • streaming-benchmarks [Java] - Benchmarks for Low Latency (Streaming) solutions including Apache Storm, Apache Spark, Apache Flink, etc.
  • flotilla [Go] - Automated message queue orchestration for scaled-up benchmarking.

Toolkit

  • akka [Scala] - toolkit and runtime for building highly concurrent, distributed, and resilient message-driven application on the JVM.
  • pulsar [Python] - Actor based event driven concurrent framework for Python.
  • aeron [Java/C++] - efficient reliable unicast and multicast message transport.
  • StreamFlow [Java] - stream processing tool designed to help build and monitor processing workflows.
  • samza-luwak [Java] - uses Luwak, a stored-query engine built on Lucene, to implement full-text search on streams.
  • Turbine [Java] - tool for aggregating streams of Server-Sent Event (SSE) JSON data into a single stream.

Readings

  1. In-Stream Big Data Processing
  2. The world beyond batch: Streaming 101 by Tyler Akidau.
  3. Real Time Analytics: Algorithms and Systems (VLDB 2015)

License

Creative Commons License

Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License