Arc

Programming language for data stream analysis.

Requirements

OCaml (and dune), Rust (and cargo), and C++ (and CMake and Ninja).

Examples

A basic streaming word-count application can be written in functional-style as follows:

val wordcounts = lines
  .flatmap(_.split(" "))
  .keyby(_)
  .window(
    length = 10min,
    stride = 3min
  )
  .count()

The same code can also be written using a more declarative, relational-style, syntax. This concept is borrowed from Morel and applied to streaming data.

val wordcounts =
  from
    line in lines,
    word in line.split(" ")
  keyby word
  window
    length = 10min
    stride = 3min
  reduce count
    identity 1;

Feature highlights

Statically typed with global type inference.
Parametric polymorphism (generics and rows) and ad-hoc polymorphism (type classes).
Mix of functional syntax, imperative control-flow/mutation, and relational operators.
Algebraic data types.
First-class data streams.
Complex event processing using tasks.
Window-based computation.
Low-level compilation and distributed execution.
Command-line interface for data ingestion.

Note: All features have not yet been implemented :)

Installation

git clone git@github.com:cda-group/arc.git
cd arc/
git submodule update --init --recursive
./build

Documentation

Project Structure

arc-lang - A compiler for Arc-Lang.
arc-mlir - An optimizer for Arc-Lang.
arc-runtime - A local runtime which supports the execution of Arc-Lang programs.
arc-python - A Python library for writing Arc-Lang applications.

Related Projects

arcon - A distributed runtime which will support execution of Arc-Lang.
kompact - A component-actor middleware which Arc-Runtime and Arcon are both implemented in.

Other

Arc-Lang ain't done until the fat lady sings. - Peter Van-Roy

XinYao1994/arc

Arc