demux-js

Demux is a backend infrastructure pattern for sourcing blockchain events to deterministically update queryable datastores and trigger side effects. This library serves as a reference implementation of that pattern for use with Node applications.

Installation

# Using yarn
yarn add demux

# Using npm
npm install demux --save

Overview

Taking inspiration from the Flux Architecture pattern and Redux, Demux was born out of the following qualifications:

A separation of concerns between how state exists on the blockchain and how it is queried by the client front-end
Client front-end not solely responsible for determining derived, reduced, and/or accumulated state
Ability for blockchain events to trigger new transactions, as well as other side effects outside of the blockchain
The blockchain as the single source of truth for all application state

Separated Persistence Layer

Storing data in indexed state on blockchains can be useful for three reasons: decentralized consensus of computation results, usage of state from within other blockchain computations, and for retrieval of state for use in client front-ends. When building more complicated front-ends, you run into a few problems when retrieving directly from indexed blockchain state:

The query interface used to retrieve the indexed data is limited. Complex data requirements can mean you either have to make an excess number of queries and process the data on the client, or you must store additional derivative data on the blockchain itself.
Scaling your query load means creating more blockchain endpoint nodes, which can be very expensive.

Demux solves these problems by off-loading queries to any persistence layer that you want. As blockchain events happen, your chosen persistence layer is updated by updater functions, which deterministically process an array of Action objects. The persistence layer can then be queried by your front-end through a suitable API (for example, REST or GraphQL).

This means that we can separate our concerns: for data that needs decentralized consensus of computation or access from other blockchain events, we can still store the data in indexed blockchain state, without having to worry about tailoring to front-end queries. For data required by our front-end, we can pre-process and index data in a way that makes it easy for it to be queried, in a horizontally scalable persistence layer of our choice. The end result is that both systems can serve their purpose more effectively.

Side Effects

Since we have a system for acting upon specific blockchain events deterministically, we can utilize this system to manage non-deterministic events as well. These effect functions work almost exactly the same as updater functions, except they run asynchronously, are not run during replays, and modifying the deterministic datastore is off-limits. Examples include: signing and broadcasting a transaction, sending an email, and initiating a traditional fiat payment.

Single Source of Truth

There are other solutions to the above problems that involve legacy persistence layers that are their own sources of truth. By deriving all state from the blockchain, however, we gain the following benefits:

If the accumulated datastore is lost or deleted, it may be regenerated by replaying blockchain actions
As long as application code is open source, and the blockchain is public, all application state can be audited
No need to maintain multiple ways of updating state (submitting transactions is the sole way)

Data Flow

Client sends transaction to blockchain
Action Watcher invokes Action Reader to check for new blocks
Action Reader sees transaction in new block, parses actions
Action Watcher sends actions to Action Handler
Action Handler processes actions through Updaters and Effects
Actions run their corresponding Updaters, updating the state of the Datastore
Actions run their corresponding Effects, triggering external events
Client queries API for updated data

Usage

This library provides the following classes:

AbstractActionReader: Abstract class used for implementing your own Action Readers
AbstractActionHandler: Abstract class used for implementing your own Action Handlers
BaseActionWatcher: Base class that implements a ready-to-use Action Watcher

View full API documentation here.

Class Implementations

Repository	Description
EOSIO / demux-js-eos *	Action Reader implementations for EOSIO blockchains
EOSIO / demux-js-postgres *	Action Handler implementation for Postgres databases

* Officially supported by Block.one

To get your project listed, add it here and submit a PR!

Example

// Let's read from an EOS node
const { NodeosActionReader } = require("demux-eos")

// Assuming you've created your own subclass of AbstractActionHandler
const MyActionHandler = require("./MyActionHandler")

// Ties everything together in a polling loop
const  { BaseActionWatcher } = require("demux")

// Import Updaters and Effects, which are arrays of objects:
// [ { actionType: string, (updater|effect): function }, ... ] 
const updaters = require("./updaters")
const effects = require("./effects")

const actionReader = new NodeosActionReader(
  "http://some-nodeos-endpoint:8888", // Locally hosted node needed for reasonable indexing speed
  12345678, // First actions relevant to this dapp happen at this block
)

const actionHandler = new MyActionHandler(
  updaters,
  effects,
)

const actionWatcher = new BaseActionWatcher(
  actionReader,
  actionHandler,
  250, // Poll at twice the block interval for less latency
)

actionWatcher.watch() // Start watch loop

For more complete examples, see the examples directory.

tianyangj/demux-js