ShareDB

ShareDB is a realtime database backend based on Operational Transformation (OT) of JSON documents. It is the realtime backend for the DerbyJS web application framework.

For questions, discussion and announcements, join the ShareJS mailing list or check the FAQ.

Please report any bugs you find to the issue tracker.

Features

Realtime synchronization of any JSON document
Concurrent multi-user collaboration
Synchronous editing API with asynchronous eventual consistency
Realtime query subscriptions
Simple integration with any database - MongoDB, PostgresQL (experimental)
Horizontally scalable with pub/sub integration
Projections to select desired fields from documents and operations
Middleware for implementing access control and custom extensions
Ideal for use in browsers or on the server
Reconnection of document and query subscriptions
Offline change syncing upon reconnection
In-memory implementations of database and pub/sub for unit testing

Example apps

Simple app demonstrating realtime sync

Leaderboard app demonstrating live queries

Data model

In ShareDB's view of the world, every document has 3 properties:

version - An incrementing number starting at 0
type - An OT type. OT types are defined in share/ottypes. Documents which don't exist implicitly have a type of null.
data - The actual data that the document contains. This must be pure acyclic JSON. Its also type-specific. (JSON type uses raw JSON, text documents use a string, etc).

ShareDB implicitly has a record for every document you can access. New documents have version 0, a null type and no data. To use a document, you must first submit a create operation, which will set the document's type and give it initial data. Then you can submit editing operations on the document (using OT). Finally you can delete the document with a delete operation. By default, ShareDB stores all operations forever - nothing is truly deleted.

Server API

Initialization

First, create a ShareDB server instance:

var ShareDB = require('sharedb');
var share = new ShareDB(options);

Options

options.db (instance of ShareDB.DB) Store documents and ops with this database adapter. Defaults to ShareDB.MemoryDB().
options.pubsub (instance of ShareDB.PubSub) Notify other ShareDB processes when data changes through this pub/sub adapter. Defaults to ShareDB.MemoryPubSub().

Database Adapters

ShareDB.MemoryDB, backed by a non-persistent database with no queries
ShareDBMongo, backed by a real Mongo database and full query support
ShareDBMingoMemory, backed by a non-persistent database supporting most Mongo queries. Useful for faster testing of a Mongo-based app.
ShareDBPostgres, backed by PostgresQL. No query support.

Pub/Sub Adapters

ShareDB.MemoryPubSub can be used with a single process
ShareDBRedisPubSub can be used with multiple processes using Redis' pub/sub mechanism

Community Provided Pub/Sub Adapters

wsbus

Listening to WebSocket connections

var WebSocketJSONStream = require('websocket-json-stream');

// 'ws' is a websocket server connection, as passed into
// new (require('ws').Server).on('connection', ...)
var stream = new WebSocketJSONStream(ws);
share.listen(stream);

For transports other than WebSockets, expose a duplex stream that writes and reads JavaScript objects. Then pass that stream directly into share.listen.

Middlewares

Middlewares let you hook into the ShareDB server pipeline. In middleware code you can read and also modify objects as they flow through ShareDB. For example, sharedb-access uses middlewares to implement access control.

share.use(action, fn) Register a new middleware.

action (String) One of:
- 'connect': A new client connected to the server.
- 'op': An operation was loaded from the database.
- 'doc': A snapshot was loaded from the database.
- 'query': A query is about to be sent to the database
- 'submit': An operation is about to be submited to the database
- 'apply': An operation is about to be applied to a snapshot before being committed to the database
- 'commit': An operation was applied to a snapshot; The operation and new snapshot are about to be written to the database.
- 'after submit': An operation was successfully submitted to the database.
- 'receive': Received a message from a client
- (ZW) 'transform': About to catch up an operation submitted against an old document version (by transforming it against all the intervening operations up until the current document version).
fn (Function(request, callback)) Call this function at the time specified by action. request contains a subset of the following properties, as relevant for the action:
- action: The action this middleware is handing
- agent: An object corresponding to the server agent handing this client
- req: The HTTP request being handled
- collection: The collection name being handled
- id: The document id being handled
- query: The query object being handled
- op: The op being handled

Projections

ShareDB supports exposing a projection of a real collection, with a specified (limited) set of allowed fields. Once configured, the projected collection looks just like a real collection - except documents only have the fields you've requested. Operations (gets, queries, sets, etc) on the fake collection work, but you only see a small portion of the data.

addProjection(name, collection, fields) Configure a projection.

name The name of the projected collection.
collection The name of the existing collection.
fields A map (object) of the allowed fields in documents.
- Keys are field names.
- Values should be true.

For example, you could make a users_limited projection which lets users view each other's names and profile pictures, but not password hashes. You would configure this by calling:

share.addProjection('users_limited', 'users', { name:true, profileUrl:true });

Note that only the JSON0 OT type is supported for projections.

Shutdown

share.close(callback) Closes connections to the database and pub/sub adapters.

Client API

The client API can be used from either Node or a browser. First, get a ShareDB.Connection object by connecting to the ShareDB server instance:

From Node:

// `share` should be a ShareDB server instance
var connection = share.connect();

To use ShareDB from a browser, use a client bundler like Browserify or Webpack. The following code connects to the ShareDB server instance over WebSockets:

var ShareDB = require('sharedb/lib/client');
var socket = new WebSocket('ws://localhost:8080');
var connection = new ShareDB.Connection(socket);

For transports other than WebSockets, create an object implementing the WebSocket specification and pass it into the ShareDB.Connection constructor.

Class: `ShareDB.Connection`

connection.get(collectionName, documentId) Get a ShareDB.Doc instance on a given collection and document ID.

connection.createFetchQuery(collectionName, query, options, callback) connection.createSubscribeQuery(collectionName, query, options, callback) Get query results from the server. createSubscribeQuery also subscribes to changes. Returns a ShareDB.Query instance.

query (Object) A descriptor of a database query with structure defined by the database adapter.
callback (Function) Called with (err, results) when server responds, or on error.
options.results (Array) Prior query results if available, such as from server rendering.
options.* All other options are passed through to the database adapter.

Class: `ShareDB.Doc`

doc.type (String) The OT type of this document

doc.id (String) Unique document ID

doc.data (Object) Document contents. Available after document is fetched or subscribed to.

doc.fetch(function(err) {...}) Populate the fields on doc with a snapshot of the document from the server.

doc.subscribe(function(err) {...}) Populate the fields on doc with a snapshot of the document from the server, and fire events on subsequent changes.

doc.ingestSnapshot(snapshot, callback) Ingest snapshot data. This data must include a version, snapshot and type. This method is generally called interally as a result of fetch or subscribe and not directly. However, it may be called directly to pass data that was transferred to the client external to the client's ShareDB connection, such as snapshot data sent along with server rendering of a webpage.

doc.destroy() Unsubscribe and stop firing events.

doc.on('load', function() {...}) The initial snapshot of the document was loaded from the server. Fires at the same time as callbacks to fetch and subscribe.

doc.on('create', function(source) {...}) The document was created. Technically, this means it has a type. source will be false for ops received from the server and defaults to true for ops generated locally.

doc.on('before op'), function(op, source) {...}) An operation is about to be applied to the data. source will be false for ops received from the server and defaults to true for ops generated locally.

doc.on('op', function(op, source) {...}) An operation was applied to the data. source will be false for ops received from the server and defaults to true for ops generated locally.

doc.on('del', function(data, source) {...}) The document was deleted. Document contents before deletion are passed in as an argument. source will be false for ops received from the server and defaults to true for ops generated locally.

doc.on('error', function(err) {...}) There was an error fetching the document or applying an operation.

doc.removeListener(eventName, listener) Removes any listener you added with doc.on. eventName should be one of 'load', 'create', 'before op', 'op', 'del', or 'error'. listener should be the function you passed in as the second argument to on. Note that both on and removeListener are inherited from EventEmitter.

doc.create(data[, type][, options][, function(err) {...}]) Create the document locally and send create operation to the server.

data Initial document contents
type (OT type) Defaults to 'ot-json0', for which data is an Object
options.source Argument passed to the 'create' event locally. This is not sent to the server or other clients. Defaults to true.

doc.submitOp(op, [, options][, function(err) {...}]) Apply operation to document and send it to the server. op structure depends on the document type. See the operations for the default 'ot-json0' type. Call this after you've either fetched or subscribed to the document.

options.source Argument passed to the 'op' event locally. This is not sent to the server or other clients. Defaults to true.
(ZW) options.docVersion Manually specifies the document version to submit the operation against.
(ZW) options.opSrc, options.opSeq Manually specifies the client ID/sequence counter that serve as the "fingerprint" for the operation.

doc.del([options][, function(err) {...}]) Delete the document locally and send delete operation to the server. Call this after you've either fetched or subscribed to the document.

options.source Argument passed to the 'del' event locally. This is not sent to the server or other clients. Defaults to true.

doc.whenNothingPending(function(err) {...}) Invokes the given callback function after

all ops submitted via doc.submitOp have been sent to the server, and
all pending fetch, subscribe, and unsubscribe requests have been resolved.

Note that whenNothingPending does NOT wait for pending model.query() calls.

Class: `ShareDB.Query`

query.ready (Boolean) True if query results are ready and available on query.results

query.results (Array) Query results, as an array of ShareDB.Doc instances.

query.extra (Type depends on database adapter and query) Extra query results that aren't an array of documents. Available for certain database adapters and queries.

query.on('ready', function() {...})) The initial query results were loaded from the server. Fires at the same time as the callbacks to createFetchQuery and createSubscribeQuery.

query.on('error', function(err) {...})) There was an error receiving updates to a subscription.

query.destroy() Unsubscribe and stop firing events.

query.on('changed', function(results) {...})) (Only fires on subscription queries) The query results changed. Fires only once after a sequence of diffs are handled.

query.on('insert', function(docs, atIndex) {...})) (Only fires on subscription queries) A contiguous sequence of documents were added to the query result array.

query.on('move', function(docs, from, to) {...})) (Only fires on subscription queries) A contiguous sequence of documents moved position in the query result array.

query.on('remove', function(docs, atIndex) {...})) (Only fires on subscription queries) A contiguous sequence of documents were removed from the query result array.

query.on('extra', function() {...})) (Only fires on subscription queries) query.extra changed.

Error codes

ShareDB returns errors as plain JavaScript objects with the format:

{
  code: 5000,
  message: 'ShareDB internal error'
}

Additional fields may be added to the error object for debugging context depending on the error. Common additional fields include collection, id, and op.

4000 - Bad request

4001 - Unknown error type
4002 - Database adapter does not support subscribe
4003 - Database adapter not found
4004 - Missing op
4005 - Op must be an array
4006 - Create data in op must be an object
4007 - Create op missing type
4008 - Unknown type
4009 - del value must be true
4010 - Missing op, create or del
4011 - Invalid src
4012 - Invalid seq
4013 - Found seq but not src
4014 - op.m invalid
4015 - Document does not exist
4016 - Document already exists
4017 - Document was deleted
4018 - Document was created remotely
4019 - Invalid protocol version
4020 - Invalid default type
4021 - Invalid client id
4022 - Database adapter does not support queries
4023 - Cannot project snapshots of this type

5000 - Internal error

The 41xx and 51xx codes are reserved for use by ShareDB DB adapters, and the 42xx and 52xx codes are reserved for use by ShareDB PubSub adapters.

5001 - No new ops returned when retrying unsuccessful submit
5002 - Missing snapshot
5003 - Snapshot and op version don't match
5004 - Missing op
5005 - Missing document
5006 - Version mismatch
5007 - Invalid state transition
5008 - Missing version in snapshot
5009 - Cannot ingest snapshot with null version
5010 - No op to send
5011 - Commit DB method unimplemented
5012 - getSnapshot DB method unimplemented
5013 - getOps DB method unimplemented
5014 - queryPollDoc DB method unimplemented
5015 - _subscribe PubSub method unimplemented
5016 - _unsubscribe PubSub method unimplemented
5017 - _publish PubSub method unimplemented
5018 - Required QueryEmitter listener not assigned

lum-ai/sharedb