Super-fast file-based sharing of Javascript objects among multiple processes.
This module maps Javascript objects into shared memory for simultaneous access by different Node processes running on the same machine. Shared memory is loaded via mmap. Object access is mediated by Boost's unordered map class so object property access are speedy.
Data is lazily loaded piece-by-piece as needed so opening even a huge file takes no time at all.
There are two modes:
A single process creates a new file which is mapped to a Javascript object. Setting properties on this object writes those properties to the file. You can read from the object within this mode but sharing an object in write-only mode with other processes is certain to result in crashes.
Open an existing file for reading. Multiple processes can safely open this file. Opening is lightning fast and only a single copy remains in memory.
If you use lengthy data values, buffers can speed things up considerably. In rough benchmarking, a 300% speedup was see when reading 20k-byte values as buffers instead of strings. For 200k-byte values, the speedup was 2000%.
Binaries are provided for OSX and Linux for various node versions (check the releases page to see which). If a binary is not provided for your platform, you will need Boost and and a C++11 compliant compiler (like GCC 4.8 or better) to build the module.
npm install mmap-object
const Shared = require('mmap-object')
const shared_object = new Shared.Create('filename')
shared_object['new_key'] = 'a string value'
shared_object.new_property = Buffer.from('a buffer value, supporting Unicode™')
shared_object['useless key'] = 0
// Erase a key
delete shared_object['useless_key']
shared_object.close()
// Read a file
const read_only_shared_object = new Shared.Open('filename')
console.log(`My value is ${read_only_shared_object.new_key}`)
console.log(`My other value is ${read_only_shared_object.new_property}`)
read_only_shared_object.close()Creates a new file mapped into shared memory. Returns an object that provides access to the shared memory. Throws an exception on error.
Arguments
path- The path of the file to createfile_size- Optional The initial size of the file in kilobytes. If more space is needed, the file will automatically be grown to a larger size. Minimum is 500 bytes. Defaults to 5 megabytes.initial_bucket_count- Optional The number of buckets to allocate initially. This is passed to the underlying Boost unordered_map. Defaults to 1024. Set this to the number of keys you expect to write.max_file_size- Optional The largest the file is allowed to grow in kilobites. If data is added beyond this limit, an exception is thrown. Defaults to 5 gigabytes.
Example
// Create a 500K map for 300 objects.
const obj = new Shared.Create('/tmp/sharedmem', 500, 300)Maps an existing file into shared memory. Returns an object that
provides read-only access to the object contained in the file. Throws
an exception on error. Any number of processes can open the same file
but only a single copy will reside in memory. Uses mmap under the
covers, so only those parts of the file that are actually accessed
will be loaded.
Arguments
path- The path of the file to open
Example
// Open up that shared file
const obj = new Shared.Open('/tmp/sharedmem')Unmaps a previously created or opened file. If the file was most
recently opened with Create(), close() will first shrink the file
to remove any unneeded space that may have been allocated.
It's important to close your unused shared files in long-running processes. Not doing so keeps shared memory from being freed.
The closing of very large objects (a few gigabytes and up) may take
some time (hundreds to thousands of milliseconds). To prevent blocking
the main thread, pass a callback to close(). The call to close()
will return immediately while the callback will be called after the
underlying munmap() operation completes. Any error will be given as
the first argument to the callback.
Example
obj.close(function (err) {
if (err) {
console.error(`Error closing object: ${err}`)
}
})The iterable protocol is supported for efficient iteration over the entire contents of the object:
const Shared = require('mmap-object')
const obj = new Shared.Open('filename')
for (let [key, value] of obj) {
console.log(`${key} => ${value}`)
}(This ES6 syntax is supported in Node 6+, for previous versions of node a more laborious syntax is necessary.)
When iterating, use isData() to tell if a particular key is real
data or one of the underlying methods on the shared object:
const obj = new Shared.Open('/tmp/sharedmem')
for (let key in obj) {
if (obj.isData(key)) { // Only show actual data
console.log(key + ': ' + obj[key])
}
}Return true if this object is currently open.
Return true if this object has been closed.
Number of bytes of free storage left in the shared object file.
The size of the storage in the shared object file, in bytes.
The number of buckets currently allocated in the underlying hash structure.
The maximum number of buckets that can be allocated in the underlying hash structure.
The average number of elements per bucket.
The current maximum load factor.
npm test
It is strongly recommended to pass in the number of keys you expect
to write when creating the object with Create. If you don't do this,
the object will resize as you fill it up. This can be a very
time-consuming process and can result in fragmentation within the
shared memory object and a larger final file size.
Object values may be only string, buffer, or number values. Attempting to set a different type value results in an exception.
Symbols are not supported as properties.
To make a new binary release:
- Edit package.json. Increment the
versionproperty. node-pre-gyp rebuildnode-pre-gyp packagenode-pre-gyp-github publishnpm publish
You will need a NODE_PRE_GYP_GITHUB_TOKEN with repo:status,
repo_deployment and public_repo access to the target repo. You'll
also need write access to the npm repo.
Set up Boost.
Set BOOST_ROOT environment variable.
bootstrap
b2 --build-type=complete