njodb is a persistent, partitioned, concurrency-controlled, Node JSON object database. Data is written to the file system and distributed across multiple files that are protected by read and write locks. By default, all methods are asynchronous and use read/write streams to improve performance and reduce memory requirements (this should be particularly useful for large databases).
Persistence - Data is saved to the file system so that it remains after the application that created it is no longer running, unlike the many existing in-memory solutions. This persistence also allows data to be made available to other applications.
Asynchronous and streaming - By default, all methods are asynchronous and non-blocking, and also use read and write streams, to make data access and manipulation efficient. Synchronous methods are also provided for those cases where they are desired or appropriate.
JSON records, not JSON files - Records are stored as individual lines of JSON objects in a file, so a read stream can be used to retrieve data rapidly, parse it in small chunks, and dispense with it when done. This removes the time and memory overhead required by solutions that store data as a single, monolithic JSON object. They must read all of that data into memory, and then parse all of it, before allowing you to use any of it.
Completely schemaless - While the JSON data itself is schemaless, it is also the case that data is not siloed into tables, or forced into collections, so the entire database, from top to bottom, is schemaless, not just the data. For a user or application, this means that there is no need to know anything about the database structure, only what data is being sought.
Balanced partitions - When inserting data, records are randomly distributed across partitions so that partition sizes are kept roughly equal, making data access times consistent. Manually resizing the database performs a similar distribution, so, as the database grows or shrinks, the partitions remain well-balanced.
Concurrency-controlled - File locks are used during read and write operations, ensuring data integrity can be maintained in a multi-user/multi-application environment. There are few, if any, existing solutions that are designed for such scenarios and that include this sort of data protection.
njodb even has its own command-line interface: check out njodb-cli.
- Install
- Test
- Introduction
- Constructor
- Database management methods
- Data manipulation methods
- Finding and fixing problematic data
npm install njodbnpm testLoad the module:
const njodb = require("njodb");Create an instance of an NJODB:
const db = new njodb.Database();Create some JSON data objects:
const data = [
{
id: 1,
name: "James",
nickname: "Good Times",
modified: Date.now()
},
{
id: 2,
name: "Steve",
nickname: "Esteban",
modified: Date.now()
}
];Insert them into the database:
db.insert(data).then(results => /* do something */ );Select some records from the database by supplying a function to find matches:
db.select(
record => record.id === 1 || record.name === "Steve"
).then(results => /* do something */ );Update some records in the database by supplying a function to find matches and another function to update them:
db.update(
record => record.name === "James",
record => { record.nickname = "Bulldog"; return record; }
).then(results => /* do something */ );Delete some records from the database by supplying a function to find matches:
db.delete(
record => record.modified < Date.now()
).then(results => /* do something */ );Delete the database:
db.drop().then(results => /* do something */ );Creates a new instance of an NJODB Database.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root |
string | Path to the root directory of the Database |
process.cwd() |
properties |
object | User-specific properties to set for the Database |
{} (see Database properties) |
If an njodb.properties file already exists in the root directory, a connection to the existing Database will be created. If the root directory does not exist it will be created. If no user-specific properties are supplied, an njodb.properties file will be created using default values; otherwise, the user-supplied properties will be merged with the default values (see Database properties below). If the data and temp directories do not exist, they will be created.
Example:
const db = new njodb.Database() // created in or connected to the current directory
const db = new njodb.Database("/path/to/some/other/place", {datadir: "mydata", datastores: 2}) // created or connected to elsewhere with user-supplied propertiesAn NJODB Database has several properties that control its functioning. These properties can be set explicitly in the njodb.properties file in the root directory; otherwise, default properties will be used. For a newly created Database, an njodb.properties file will be created using default values.
Properties:
| Name | Type | Description | Default |
|---|---|---|---|
datadir |
string | The name of the subdirectory of root where data files will be stored |
data |
dataname |
string | The file name that will be used when creating or accessing data files | data |
datastores |
number | The number of data partitions that will be used | 5 |
tempdir |
string | The name of the subdirectory of root where temporary data files will be stored |
tmp |
lockoptions |
object | The options that will be used by proper-lockfile to lock data files | {"stale": 5000, "update": 1000, "retries": { "retries": 5000, "minTimeout": 250, "maxTimeout": 5000, "factor": 0.15, "randomize": false } } |
stats
Returns statistics about the Database. Resolves with the following information:
| Name | Description |
|---|---|
root |
The path of the root directory of the Database |
data |
The path of the data subdirectory of the Database |
temp |
The path of the temp subdirectory of the Database |
records |
The number of records in the Database (the sum of the number of records in each datastore) |
errors |
The number of problematic records in the Database |
size |
The total size of the Database in "human-readable" format (the sum of the sizes of the individual datastores) |
stores |
The total number of datastores in the Database |
min |
The minimum number of records in a datastore |
max |
The maximum number of records in a datastore |
mean |
The mean (i.e., average) number of records in each datastore |
var |
The variance of the number of records across datastores |
std |
The standard deviation of the number of records across datastores |
start |
The timestamp of when the stats call started |
end |
The timestamp of when the stats call finished |
elapsed |
The amount of time in milliseconds required to run the stats call |
details |
An array of detailed stats for each datastore |
A synchronous version of stats.
grow()
Increases the number of datastores by one and redistributes the data across them.
growSync()
A synchronous version of grow.
shrink()
Decreases the number of datastores by one and redistributes the data across them. If the current number of datastores is one, calling shrink() will throw an error.
shrinkSync()
A synchronous version of shrink.
resize(size)
Changes the number of datastores and redistributes the data across them.
Parameters:
| Name | Type | Description |
|---|---|---|
size |
number | The number of datastores (must be greater than zero) |
resizeSync(size)
A synchronous version of resize.
drop()
Deletes the database, including the data and temp directories, and the properties file.
dropSync()
A synchronous version of drop.
getProperties()
Returns the properties set for the Database. Will likely be deprecated in a future version.
setProperties(properties)
Sets the properties for the the Database. Will likely be deprecated in a future version.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
properties |
object | The properties to set for the Database |
See Database properties |
insert(data)
Inserts data into the Database.
Parameters:
| Name | Type | Description |
|---|---|---|
data |
array | An array of JSON objects to insert into the Database |
Resolves with an object containing results from the insert:
| Name | Type | Description |
|---|---|---|
inserted |
number | The number of objects inserted into the Database |
start |
date | The timestamp of when the insertions began |
end |
date | The timestamp of when the insertions finished |
elapsed |
number | The amount of time in milliseconds required to execute the insert |
details |
array | An array of insertion results for each individual datastore |
insertSync(data)
A synchronous version of insert.
insertFile(file)
Inserts data into the database from a file containing JSON data. The file itself does not need to be a valid JSON object, rather it should contain a single stringified JSON object per line. Blank lines are ignored and problematic data is collected in an errors array.
Resolves with an object containing results from the insertFile:
| Name | Type | Description |
|---|---|---|
inspected |
number | The number of lines of the file inspected |
inserted |
number | The number of objects inserted into the Database |
blanks |
number | The number of blank lines in the file |
errors |
array | An array of problematic records in the file |
start |
date | The timestamp of when the insertions began |
end |
date | The timestamp of when the insertions finished |
elapsed |
number | The amount of time in milliseconds required to execute the insert |
details |
array | An array of insertion results for each individual datastore |
An example data file, data.json, is included in the test subdirectory. Among many valid records, it also includes blank lines and a malformed JSON object. To insert its data into the database:
db.insertFile("./test/data.json").then(results => /* do something */ );insertFileSync(file)
A synchronous version of insertFile.
select(selecter [, projector])
Selects data from the Database.
Parameters:
| Name | Type | Description |
|---|---|---|
selecter |
function | A function that returns a boolean that will be used to identify the records that should be returned |
projecter |
function | A function that returns an object that identifies the fields that should be returned |
Resolves with an object containing results from the select:
| Name | Type | Description |
|---|---|---|
data |
array | An array of objects selected from the Database |
selected |
number | The number of objects selected from the Database |
ignored |
number | The number of objects that were not selected from the Database |
errors |
array | An array of problematic (i.e., un-parseable) records in the Database |
start |
date | The timestamp of when the selections began |
end |
date | The timestamp of when the selections finished |
elapsed |
number | The amount of time in milliseconds required to execute the select |
details |
array | An array of selection results, including error details, for each individual datastore |
Example with projection that selects all records, returns only the id and modified fields, but also creates a new one called newID:
db.select(
() => true,
record => { return {id: record.id, newID: record.id + 1, modified: record.modified }; }
);selectSync(selecter [, projector])
A synchronous version of select.
update(selecter, updater)
Updates data in the Database.
Parameters:
| Name | Type | Description |
|---|---|---|
selecter |
function | A function that returns a boolean that will be used to identify the records that should be updated |
updater |
function | A function that applies an update to a selected record and returns it |
Resolves with an object containing results from the update:
| Name | Type | Description |
|---|---|---|
selected |
number | The number of objects selected from the Database for updating |
updated |
number | The number of objects updated in the Database |
unchanged |
number | The number of objects that were not updated in the Database |
errors |
array | An array of problematic (i.e., un-parseable) records in the Database or records that were unable to be updated |
start |
date | The timestamp of when the updates began |
end |
date | The timestamp of when the updates finished |
elapsed |
number | The amount of time in milliseconds required to execute the update |
details |
array | An array of update results, including error details, for each individual datastore |
updateSync(selecter, updater)
A synchronous version of update
delete(selecter)
Deletes data from the Database.
Parameters:
| Name | Type | Description |
|---|---|---|
selecter |
function | A function that returns a boolean that will be used to identify the records that should be deleted |
Resolves with an object containing results from the delete:
| Name | Type | Description |
|---|---|---|
deleted |
number | The number of objects deleted from the Database |
retained |
number | The number of objects that were not deleted from the Database |
errors |
array | An array of problematic (i.e., un-parseable) records in the Database or records that were unable to be deleted |
start |
date | The timestamp of when the deletions began |
end |
date | The timestamp of when the deletions finished |
elapsed |
number | The amount of time in milliseconds required to execute the delete |
details |
array | An array of deletion results, including error details, for each individual datastore |
deleteSync(selecter)
A synchronous version of delete.
aggregate(selecter, indexer [, projecter])
Aggregates data in the database.
Parameters:
| Name | Type | Description |
|---|---|---|
selecter |
function | A function that returns a boolean that will be used to identify the records that should be aggregated |
indexer |
function | A function that returns an object that creates the index by which data will be grouped |
projecter |
function | A function that returns an object that identifies the fields that should be returned |
Resolves with an object containing results from the aggregate:
| Name | Type | Description |
|---|---|---|
data |
array | An array of index objects selected from the Database |
indexed |
number | The number of records that were indexable (i.e., processable by the indexer function) |
unindexed |
number | The number of records that were un-indexable |
errors |
number | The number of problematic (i.e., un-parseable) records in the Database |
start |
date | The timestamp of when the aggregations began |
end |
date | The timestamp of when the aggregations finished |
elapsed |
number | The amount of time in milliseconds required to execute the aggregate |
details |
array | An array of selection results, including error details, for each individual datastore |
Each index object contains the following:
| Name | Type | Description |
|---|---|---|
index |
any valid type | The value of the index created by the indexer function |
count |
number | The count of records that contained the index |
data |
array | An array of aggregation objects for each field of the records returned |
Each aggregation object contains one or more of the following (non-numeric fields do not contain numeric aggregate data):
| Name | Type | Description |
|---|---|---|
min |
any valid type | Minimum value of the field |
max |
any valid type | Maximum value of the field |
sum |
number | The sum of the values of the field (undefined if not a number) |
mean |
number | The mean (i.e., average) of the values of the field (undefined if not a number) |
varp |
number | The population variance of the values of the field (undefined if not a number) |
vars |
number | The sample variance of the values of the field (undefined if not a number) |
stdp |
number | The population standard deviation of the values of the field (undefined if not a number) |
stds |
number | The sample standard deviation of the values of the field (undefined if not a number) |
An example that generates aggregates for all records and fields, grouped by state and lastName:
db.aggregate(
() => true,
record => [record.state, record.lastName]
);Another example that generates aggregates for records with an ID less than 1000, grouped by state, but for only two fields (note the non-numeric fields do not include numeric aggregate data):
db.aggregate(
record => record.id < 1000,
record => record.state,
record => { return {favoriteNumber: record.favoriteNumber, firstName: record.firstName}; }
);Example aggregate data array:
[
{
index: "Maryland",
count: 50,
aggregates: [
{
field: "favoriteNumber",
data: {
min: 0,
max: 98,
sum: 2450,
mean: 49,
varp: 833,
vars: 850,
stdp: 28.861739379323623,
stds: 29.154759474226502
}
},
{
field: "firstName",
data: {
min: "Elizabeth",
max: "William"
}
}
]
},
{
index: "Virginia",
count: 50,
aggregates: [
{
field: "favoriteNumber",
data: {
min: 0,
max: 49,
sum: 1225,
mean: 24.5,
varp: 208.25000000000003,
vars: 212.50000000000003,
stdp: 14.430869689661813,
stds: 14.577379737113253
}
},
{
field: "firstName",
data: {
min: "James",
max: "Robert"
}
}
]
}
]aggregate(selecter, indexer [, projecter])
A synchronous version of aggregate.
Many methods return information about problematic records encountered (e.g., records that are not parseable using JSON.parse(), or ones that couldn't be updated or deleted); both a count of them, as well as details about them in the details array. The objects in the details array - one for each datastore - contain an errors array that is a collection of objects about problematic records.
For un-parseable records, each error object includes the line of the datastore file where the problematic record was found as well as a copy of the record itself. With this information, if one wants to address these problematic data they can simply load the datastore file in a text editor and either correct the record or remove it. For records that couldn't be deleted or updated, each error object includes a copy of the record itself. With this information, one could make another attempt to update or delete the record(s), or otherwise handle the failure.
Here is an example of the details for a datastore that contains an un-parseable record. As you can see, the record is on the tenth line of the file, and the problem is that the lastname key name is missing an enclosing quote. Simply adding the quote fixes the record.
{
store: '/Users/jamesbontempo/github/njodb/data/data.0.json',
size: 1512464,
lines: 8711,
records: 8709,
errors: [
{
error: 'Unexpected token D in JSON at position 42',
line: 10,
data: '{"id":232,"firstName":"Robert","lastName:"Davis","state":"Illinois","birthdate":"1990-10-22","favoriteNumbers":[5,34,1],"favoriteNumber":183,"modified":1616806973645}'
}
],
blanks: 1,
created: 2021-03-27T01:20:21.562Z,
modified: 2021-03-27T01:28:32.686Z,
start: 1616808517081,
end: 1616808517124,
elapsed: 43
}