kappa-osm

Peer-to-peer OpenStreetMap database over kappa-core

Purpose

A simple and easy-to-use geographic/spatial database that works offline, and can synchronize with other instances of the database using a variety of methods, internet and non (local wifi, USB keys, bluetooth, and more).

Current Status

MOSTLY STABLE

Several upstream modules are using this now, and are being integrated into "real" apps. Expect minimal breaking changes going forward.

If you're interested in this project, leave an issue and share a bit about what you're building & how we might collaborate!

Usage

var kappa = require('kappa-core')
var ram = require('random-access-memory')
var memdb = require('memdb')
var Osm = require('kappa-osm')

var osm = Osm({
  core: kappa(ram, { valueEncoding: 'json' }),
  index: memdb(),
  storage: function (name, cb) { cb(null, ram()) }
})

var node = {
  type: 'node',
  lat: '-12.7',
  lon: '1.3',
  tags: { feature: 'water fountain' },
  changeset: 'abcdef'
}

osm.create(node, function (err, node) {
  if (err) return console.error(err)

  console.log('created node with id', node.id)

  osm.query([1,-13,2,-11], function (err, nodes) {
    if (err) console.error(err)
    else console.log(nodes)
  })
})

outputs

created node with id 58261217205dc19b
[
  {
    id: '58261217205dc19b',
    type: 'node',
    lat: '-12.7',
    lon: '1.3',
    tags: { feature: 'water fountain' },
    changeset: 'abcdef' },
    version: '366212350b5996f944df9df25e679a98545bdac98f507a06f493d167ff9d5f14@0',
    links: [],
    deviceId: '366212350b5996f944df9df25e679a98545bdac98f507a06f493d167ff9d5f14',
    authorId: '366212350b5996f944df9df25e679a98545bdac98f507a06f493d167ff9d5f14'
  }
]

API

var Osm = require('kappa-osm')

var osm = Osm(opts)

Expected opts include:

core: a kappa-core instance
index: a levelup instance
storage: a function (name, cb) {} that should provide a random-access-storage instance to its callback cb

osm.create(element, cb)

Create the new OSM element element and add it to the database. The resulting element, populated with the id, version, and authorId, and deviceId fields, is returned by the callback cb.

authorId is the original author's deviceId, while deviceId represents the device that most recently edited the element.

osm.get(id, cb)

Fetch all of the newest OSM elements with the ID id. cb is called with an array of OSM elements.

The reason an array is returned is because of the distributed nature of the database: in the case that multiple peers modify an element prior to sync'ing their databases with each other, there would be multiple latest elements ("heads") for that ID.

If no elements with id exist, an empty array is returned.

osm.getByVersion(version[, opts], cb)

Fetch a specific OSM element by its version string. Returns null if not found, otherwise the single element.

osm.put(id, element, [opts, ]cb)

Update an existing element with ID id to be the OSM element element. The new element should have all fields that the OSM element would have. The type of the element cannot be changed.

Updates work by replacing old heads (latest versions) with a new version. This works by "linking" back to the version names of all previous heads you want to replace. This happens automatically, but if an array of versions are passed into opts.links, those elements will be replaced with this newer version instead of the default current heads.

cb is called with the new element, including id, version, and deviceId properties.

osm.del(id, value, cb)

Marks the element id as deleted. Since all data is append-only in the database, this does not actually delete data, but instead writes a brand new version of the document with { deleted: true } set on it.

Deleted ways, nodes, and relations are all still returned by the query API. The nodes of a deleted way are not included in the results.

osm.batch(ops, cb)

Create and update many elements atomically. ops is an array of operations (objects) describing the elements to be added or updated or deleted.

{
  type: 'put|del',
  id: 'id',
  value: { /* element */ }
}

If no id field is set, the element is created, otherwise it is updated with the element value.

An operation type of 'put' inserts a new element or modifies an existing one, while a type of'del' will mark the element as deleted.

Currently, doing a batch insert skips many validation checks in order to be as fast as possible.

TODO: means to enable validation + error reporting / atomic write

var rs = osm.query(bbox[, opts] [, cb])

Retrieves all nodes, ways, and relations within the bounding box bbox.

bbox is expected to be an array of the form [WEST, SOUTH, EAST, NORTH]. Latitude (north/south) runs between (-85, 85), and longitude (west/east) between (-180, 180).

A callback parameter cb is optional. If given, it will be called as cb(err, elements). If not provided, a Readable stream will be returned that can be read from as elements are emitted. The distinction between the two is that the callback will buffer all results before they are returned, but the stream will emit results as they arrive, resulting in much less buffering. This can make a large impact on memory use for queries on large datasets.

Elements are returned as governed by the query algorithm outlined by the OSM v0.6 API:

All nodes that are inside a given bounding box and any relations that reference them.
All ways that reference at least one node that is inside a given bounding box, any relations that reference them (the ways), and any nodes outside the bounding box that the ways may reference.
All relations that reference one of the nodes, ways or relations included due to the above rules. (This does not apply recursively; meaning that elements referenced by a relation are not returned by virtue of being in that relation.)

Accepted opts include:

opts.observations (boolean): whether to include type === 'observation' objects as well as regular OSM types.

var rs = osm.refs(id[, cb])

Fetch a list of all OSM elements that refer to the element with ID id. This captures

elements with a changeset field
all nodes referenced by a way's nodes field
all nodes, ways, and relations referenced by a relation's members field

TODO: this could be made clearer -- maybe an example?

A callback parameter cb is optional. If given, it will be called as cb(err, results). If not provided or set to null, a Readable stream will be returned that can be read from as results are ready.

Objects of the following form are returned:

{
  id: '...',
  version: '...'
}

var r = osm.history(opts)

Return a readable stream r of all the documents in the db sorted by timestamp or created_at (for observations). By default, returns least recent documents first.

The following options are accepted via the opts parameter:

opts.type(boolean) - additionally filter results by type as a string
opts.id (boolean) - only show results for a single string id
opts.reverse (boolean) - when true, provide results from most to least recent
opts.lt, opts.lte, opts.gt, opts.gte (string) - lexicographic sorting options

TODO: clarify how lt/lte/gt/gte work

There is a separate index for filtering by type and ID each, so queries should be fast. Filtering by ID or type are exclusive options.

The lexicographic sorting options operate on timestamp/created_at keys which are in ISO 8601 format, as you could get from .toISOString():

> new Date().toISOString()
'2018-09-14T23:07:53.862Z'

var t = osm.byType(type)

Returns a readable stream t of all documents in the database of type type. Only the latest documents are returned, not historic data.

var stream = osm.replicate(isInitiator, [opts])

Create a duplex replication stream, that you can pipe into another kappa-osm's instance's replication stream. The stream ends once all map data is exchanged between the two peers.

Ensure that isInitiator to true to one side, and false on the other.

opts are passed in to multifeed's API of the same name.

osm.on('error', function (err) {})

Event emitted when an error within kappa-osm has occurred. This is very important to listen on, lest things suddenly seem to break and it's not immediately clear why.

Architecture

Document Format

Documents (OSM elements, observations, etc) have a common format:

  {
    id: String,
    type: String,
    lat: String,
    lon: String,
    tags: Object,
    changeset: String,
    links: Array<String>,
    version: String,
    deviceId: String,
    authorId: String
  }

Note: authorId is the original author's deviceId, while deviceId represents the device that most recently edited the element.

TODO: talk about forking data & forking architecture*

Install

With npm installed, run

$ npm install kappa-osm

License

ISC

baal-lgln/kappa-osm