/arc

Programming Language for Continuous Deep Analytics

Primary LanguageMLIR

Arc

Programming language for data stream analysis.

Requirements

OCaml (and dune), Rust (and cargo), and C++ (and CMake and Ninja).

Examples

A basic streaming word-count application can be written in functional-style as follows:

val wordcounts = lines
  .flatmap(_.split(" "))
  .keyby(_)
  .window(
    length = 10min,
    stride = 3min
  )
  .count()

The same code can also be written using a more declarative, relational-style, syntax. This concept is borrowed from Morel and applied to streaming data.

val wordcounts =
  from
    line in lines,
    word in line.split(" ")
  keyby word
  window
    length = 10min
    stride = 3min
  reduce count
    identity 1;

Feature highlights

  • Statically typed with global type inference.
  • Parametric polymorphism (generics and rows) and ad-hoc polymorphism (type classes).
  • Mix of functional syntax, imperative control-flow/mutation, and relational operators.
  • Algebraic data types.
  • First-class data streams.
  • Complex event processing using tasks.
  • Window-based computation.
  • Low-level compilation and distributed execution.
  • Command-line interface for data ingestion.

Note: All features have not yet been implemented :)

Installation

git clone git@github.com:cda-group/arc.git
cd arc/
git submodule update --init --recursive
./build

Documentation

Project Structure

  • arc-lang - A compiler for Arc-Lang.
  • arc-mlir - An optimizer for Arc-Lang.
  • arc-runtime - A local runtime which supports the execution of Arc-Lang programs.
  • arc-python - A Python library for writing Arc-Lang applications.

Related Projects

  • arcon - A distributed runtime which will support execution of Arc-Lang.
  • kompact - A component-actor middleware which Arc-Runtime and Arcon are both implemented in.

Other

Arc-Lang ain't done until the fat lady sings. - Peter Van-Roy