Goldrush is a small Erlang app that provides fast event stream processing

Event processing compiled to a query module

per module private event processing statistics
query module logic can be combined for any/all filters
query module logic can be reduced to efficiently match event processing

Complex event processing logic

match input events with greater than (gt) logic
match input events with less than (lt) logic
match input events with equal to (eq) logic
match input events with wildcard (wc) logic
match input events with notfound (nf) logic
match no input events (null blackhole) logic
match all input events (null passthrough) logic

Handle output events

Once a query has been composed the output action can be overriden with one or more erlang functions. The functions will be applied to each output event from the query.

Handle low latency retrieval of compile-time stored values.

Values stored are also provided to functions called on event output.
Handle job execution and timing which can also get values stored
create input events that include runtime on successful function executions.

Handle fastest lookups of stored values.

provide state storage option to compile, caching the values in query module.

Usage

To use goldrush in your application, you need to define it as a rebar dep or include it in erlang’s path.

Before composing modules, you’ll need to define a query. The query syntax matches any number of `{erlang, terms}’ and is composed as follows:

Simple Logic

Simple logic is defined as any logic matching a single event filter

Select all events where ‘a’ exists and is greater than 0.

glc:gt(a, 0).

Select all events where ‘a’ exists and is greater than or equal to 0.

glc:gte(a, 0).

Select all events where ‘a’ exists and is equal to 0.

glc:eq(a, 0).

Select all events where ‘a’ exists and is not equal to 0.

glc:neq(a, 0).

Select all events where ‘a’ exists and is less than 0.

glc:lt(a, 0).

Select all events where ‘a’ exists and is less than or equal to 0.

glc:lte(a, 0).

Select all events where ‘a’ exists.

glc:wc(a).

Select all events where ‘a’ does not exist.

glc:nf(a).

Select no input events. User as a black hole query.

glc:null(false).

Select all input events. Used as a passthrough query.

glc:null(true).

Combined Logic

Combined logic is defined as logic matching multiple event filters

Select all events where both ‘a’ AND ‘b’ exists and are greater than 0.

glc:all([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ OR ‘b’ exists and are greater than 0.

glc:any([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ AND ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:all([glc:gt(a, 1), glc:lt(b, 2)]).

Select all events where ‘a’ OR ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:any([glc:gt(a, 1), glc:lt(b, 2)]).

Reduced Logic

Reduced logic is defined as logic which can be simplified to improve efficiency.

Select all events where ‘a’ is equal to 1, ‘b’ is equal to 2 and ‘c’ is equal to 3 and collapse any duplicate logic.

glc_lib:reduce(
    glc:all([
        glc:any([glc:eq(a, 1), glc:eq(b, 2)]),
        glc:any([glc:eq(a, 1), glc:eq(c, 3)])])).

The previous example will produce and is equivalent to:

glc:all([glc:eq(a, 1), glc:eq(b, 2), glc:eq(c, 3)]).

Composing Modules

All query modules must be compiled before use

To compose a module you will take your Query defined above and compile it.

glc:compile(Module, Query).
glc:compile(Module, Query, State).
glc:compile(Module, Query, State, ResetStatistics).

At this point you will be able to handle an event using a compiled query.

Begin by constructing an event list.

Event = gre:make([{'a', 2}], [list]).

Now pass it to your query module to be handled.

glc:handle(Module, Event).

Handling output events

You can override the output action with an erlang function.

Write all input events as info reports to the error logger.

glc:with(glc:null(true), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is less than 5 as info reports to the error logger.

glc:with(glc:lt(error_level, 5), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is 3 or 5 as info reports to the error logger.

glc:any([
  glc:with(glc:lt(error_level, 3), fun(E) ->
      error_logger:info_report(gre:pairs(E)) end),
  glc:with(glc:lt(error_level, 5), fun(E) ->
      error_logger:info_report(gre:pairs(E)) end)]).

To compose a module with state data you will add a third argument (orddict).

glc:compile(Module, Query, [{stored, value}]).

Return the stored value in this query module.

{ok, value} = glc:get(stored).

Return all stored values in this query module.

[...] = Module:get().

Composing Modules with stored data

You can create query modules with local state to compare to event data in `with’ and `run’

To compose a module with state data you will add a third argument (orddict).

glc:compile(Module, Query, [{stored, value}]).

Accessing stored data in constant time

You can use query modules in a way similar to mochiglobal

Return the stored value in this query module.

{ok, value} = glc:get(stored).

Job processing with composed modules

You can use query modules to execute jobs, if the job errors or not, process an event.
`with’ is similar to `run’, the main difference is additional statistics and execution order
when a job completes in error, the event data will contain an additional {error, _} item

To execute a job through the query module, inputting an event on success.

Event = gre:make([{'a', 2}], [list]).
{ExecutionTime, Result}= glc:run(Module, fun(Event, State) ->
    %% do not end with {error, _} or throw an exception 
end, Event).

Event Processing Statistics

Return the number of input events for this query module.

glc:input(Module).

Return the number of output events for this query module.

glc:output(Module).

Return the number of filtered events for this query module.

glc:filter(Module).

Job Processing Statistics

Return the number of job runs for this query module.

glc:job_run(Module).

Return the number of job errors for this query module.

glc:job_error(Module).

Return the number of job inputs for this query module.

glc:job_input(Module).

Return the amount of time jobs took for this query module.

glc:job_time(Module).

Some Tips & Tricks

This is really just a drop in the bucket.

Return the average time jobs took for this query module.

glc:job_time(Module) / glc:job_input(Module) / 1000000.

Return the query combining the conditional logic of multiple modules

glc_lib:reduce(glc:all([Module1:info('query'), Module2:info('query')]).

Return all statistics from this query module.

glc:info(Module).

Build

$ ./rebar compile

$ make

CHANGELOG

0.1.9

Add support for running jobs

0.1.8

Add support for not equal

0.1.7

Support multiple functions specified using `with/2`
Add support for greater than or less than operators
Add state storage option for output events or lookup

0.1.7

Add job execution and timings
Add state storage option

0.1.7

Add job execution and timings
Add state storage option

0.1.6

Add notfound event matching

0.1.5

Rewrite to make highly crash resilient
- per module supervision
- statistics data recovery
Add wildcard event matching
Add reset counters

bob403/goldrush

Event processing compiled to a query module

Complex event processing logic

Handle output events

Handle low latency retrieval of compile-time stored values.

Handle fastest lookups of stored values.

Usage

Simple Logic

Combined Logic

Reduced Logic

Composing Modules

Handling output events

Composing Modules with stored data

Accessing stored data in constant time

Job processing with composed modules

Event Processing Statistics

Job Processing Statistics

Some Tips & Tricks

Build

CHANGELOG