Lucasczm/nun-db

A realtime database written in rust

NunDB

What is Nun DB

Nun DB is an open source real-time database, made to be fast, light and easy to use.

Nun DB is written to be memory safe and horizontal(soon) scalable.

We believe we can keep Nun DB simple and at the same time it can power lots of different apps and use cases.

Examples Use cases

Checkout our examples of integration and see what we can do for your app

• Vue.js + Nun DB jsfiddle
• React.js Realtime Todo app

Installations

Docker

Running NunDB using Docker is likely the fastest method. The simplest steps to run NunDB with Docker involve running a container, as shown in the following example.

#Change user-name to the admin you want, user-pwd to the admin pwd you want, change /tmp to the directory where you want to store your data in the host computer... for dev /tmp works fine
docker run --env NUN_USER=user-name --env NUN_PWD=user-pwd --env NUN_DBS_DIR="/data" --mount type=bind,source="/tmp",target=/data -it --rm -p 3013:3013 -p 3012:3012 -p 3014:3014 --name nun-test mateusfreira/nun-db

Docker-compose

version: '3'
services:
  nun-db:
    image: "mateusfreira/nun-db"
    ports:
      - "3012:3012" # Web socket
      - "3013:3013" # Http
      - "3014:3014" # Socket
    environment:
      - NUN_DBS_DIR=/nun_data 
      - NUN_USER=mateus
      - NUN_PWD=mateus
    volumes:
        - /tmp/data/nun_db/:/nun_data

Note that the "/tmp/data/nun_db/" is a directory in the machine where the nun-db is running so you may need to create the directory for that use the command mkdir /tmp/data/nun_db/.

Create the sample DB:

# First connect to the running container
docker exec -it nun-test  /bin/sh

# Then execute the command to create the database
nun-db -u user-name -p user-pwd create-db -d sample -t sample-pwd

# You should see something like
Response "valid auth\n;create-db success\n"

Once done, you now have NunDB running in your docker and exposing all the ports for use: HTTP (3013), WebSocket (3012), and Socket (3014) you are ready to use.

How-tos

• Real-time Medical Image Collaboration POC Made Easy with OHIF and NunDB

• How to create users with different permission levels in NunDB

• How to Building a Trello-like React/Redux App with Nun-DB with offline and conflict resolution features

• How-to Make Redux TodoMVC Example a real-time multiuser with NunDB in 10 steps

• How to create your simple version of google analytics real-time using NunDB

• NunDb How to backup one or all databases

• How to snapshot NunDB databases from memory to disk

# TLDR version
nun-db --user $NUN_USER  -p $NUN_PWD --host "https://http.nundb.org" exec "use-db $DB_NAME $DB_TOKEN; snapshot"

• Introduction to manage conflicts in NunDB

Technical documentations

A fast-to-sync/search and space-optimized replication algorithm written in rust, The NunDB data replication model

Leader election in rust the journey towards implementing NunDB leader election

Integration tests in rust a multi-process test example

Writing a prometheus exporter in rust from idea to grafana chart

Why does NunDB we have a Debug command?

Diagram

                                      .------------------------------------------.
                                      |                     |                    |              .---------------.
                                      |       http_ops      |                    |------------->|  Disck        |      
                                      |                     |   disk_ops         |------------->|               |      
                                      |_____________        |                    |              .---------------.
                                      |            |        ._________________.__|                                     
                                      |            |        |                 |  |                                     
                                      |            |        | replication_ops |  |                                     
                                      |   tcp_ops   \_______+--------+--------+  |                                     
                                      |             |       |        |           |                                     
.----------------.                    |             |       |        |           |      .---------------. 
|   Frontends/   | ---text----------> |_____________|parse <.> core  | db_ops    |----->|   Memory      | 
|  Clients       | <---text---------- |             |       |        |           |<-----| Repository    | 
.----------------.                    |             |       |        |           |      |               | 
                                      |             \_______|_______/________.   |      ._______________.
                                      |    ws_ops           |                |   |                                    
                                      |                     |   election_ops |   |                                    
                                      |_____________________._______________/____|                                    
                                      |                                          |                                    
                                      |                                          |                                    
                                      |                 monitoring               |                                    
                                      |                                          |                                    
                                      |                                          |                                    
                                      .------------------------------------------.                                    

Connectors

• Http Port: 3013

• Socket Port: 3014

• Web Socket Port: 3012

Having any issue or needs help?

Open an issue I will follow as soon as possible.

Want to use it in production ?

Talk to me @mateusfreira I will help you get it to the scale you need.

Nun Query language (NQL)

Auth

Context

• Require admin auth
• Require db auth
• Replicate? How? ()

e.g:

auth $user $pwd

UseDb

Context

• Require admin auth
• Require db auth
• Replicate? How? ()

e.g:

use-db $db-name $db-pwd

Alias

use

e.g:

use $db-name $db-pwd

e.g:

use-db $db-bame $user-name $token

CreateDb

Context

• Require admin auth
• Require db auth
• Replicate? How? (create-db)
• Register Oplog? How? (Update) Creates a new database

e.g.,

# create simple db
create-db test test-pwd;
response: 
empty

# create db with arbiter conlifct resolution strategy
create-db test test-pwd arbiter;
response:
empty

Get

Context

• Require admin auth
• Require db auth
• Replicate? How? ()

Set

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate)

SetSafe

Changes the value of a key, guarantees consistency by version Soon this will be the default way to set values to a key, since we are moving to a leader less replication model.

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate) Examples:

set-safe $key $version $value
set-safe name 10 Mateus # Sets key name to mateus if version is equal or minor than 

Remove

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate)

Snapshot $reclaim_space(true|false) $db_names_pipe_separated (Empty means will snapshot the current database only)

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-snapshot)

E.g:

Snapshot the databases to disk, faster method will store only the difference will use more disk space.

snapshot false db1|db2|db3|db4

Snapshot the database to disk, slower method will store all data again to disk and reorganize the values file

snapshot true db1|db2|db3|db4

Snapshot the database to disk only the current database

snapshot true

UnWatch

Context

• Require admin auth
• Require db auth
• Replicate? How?

UnWatchAll

Context

• Require admin auth
• Require db auth
• Replicate? How?

Watch

Context

• Require admin auth
• Require db auth
• Replicate? How?

Keys

Return the list of keys for the database.

Context

• Require admin auth
• Require db auth
• Replicate? How?

e.gs:

• Return all keys keys
• Return keys starting with name keys name*
• Return keys ending with name keys *name
• Return keys containing with name keys name

Alias

ls

SetPrimary

Context

• Require admin auth
• Require db auth
• Replicate? How?

ElectionWin

Context

• Require admin auth
• Require db auth
• Replicate? How?

Join

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-join)

ReplicateSince

Context

• Require admin auth
• Require db auth
• Replicate? How?

Increment

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-increment)
• Register Oplog? How? (Update)

Increments an integer key value, if the key does not exist start the value with 0, if the key is integer returns an error e.gs: Increments in 1

increment visits

Increments by 10

increment visits 10

Acknowledge

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-increment)
• Register Oplog? How? (Update)

Internally used to acknowledge messages processed by the replicas

e.g

  ack 1 replica1:181

ClusterState

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-increment)
• Register Oplog? How? (Update)

Returns the cluster state, useful for debugging or admin purposes

e.gs

# request
cluster-state;
response: 
cluster-state  127.0.0.1:3017:Primary, 127.0.0.1:3018:Secoundary,

MetricsState

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-increment)
• Register Oplog? How? (Update)

Returns the oplog and query metrics state, useful for debugging or admin proposes

e.gs

# request
metrics-state;
response: 
metrics-state pending_ops: 0, op_log_file_size: 0, op_log_count: 0,replication_time_moving_avg: 0.0, get_query_time_moving_avg: 0.0

Debug

Context

• Require admin auth
• Require db auth
• Replicate? How? (replicate-increment)
• Register Oplog? How? (Update)

The debug command holds admin queries for NunDB, such as, checking the messages that are pending replication from a specific node in the cluster.

e.gs

debug pending-ops
# result
pending-ops #list-of-pending-ops#

debug pending-conflicts
# result (Must be connected to a database with use *** ***)
pending-conflicts #list-of-pending-conflicts#

debug list-dbs
# result (List the name and the strategy of the dbs one on each line)
dbs-list
sample : none
vue : none
test : arbiter
analitcs-blog : none
$admin : newer

debug force-election
# Forces run an election immediately, which is useful if some node in the cluster is not responsive or to debug latency problems between nodes. Elections in NunDB are predictable and, unless the primary node is slow, the primary should not change even if you force an election.

Arbiter

Context

• Require admin auth
• Require db auth
• Replicate? How? ()
• Register Oplog? How? (no)

With the arbiter command one client register it self as an arbiter for conflicts, this client must be connected to the primary database. Internally this command is equivalent to watching the key $conflict.

e.gs

arbiter
#result
ok|error

CreateUser

Context

• Require admin auth
• Require db auth
• Replicate? How? (Set command to security key)
• Register Oplog? How? (As key value)

Creates a user for the selected user e.gs

create-user $user $token

SetPermission

Context

• Require admin auth
• Require db auth
• Replicate? How? (Set command to security key)
• Register Oplog? How? (As key value)

Creates a user for the selected user e.gs

set-permissions $user_name $permissions $key_pattern|$permissions $key_pattern;

set-permissions foo rw test-*|rwi count;
# Grant permission to user called foo to read and write keys starting with test-, and read write and increment count key.
# Permissions latter
# r = Read,
# w = Write,
# i = Increment,
# x = Remove,

Special keys

All special keys will have a $ symbol in the first letter of the name.

$connections

Count the number of connections to a database.

$connections

$conflicts

• Key used to register client as arbiter for conflict resolution

Secure keys

• All keys prefixed with $$ will be considered secure by NunDB and will only allow database admin authentication to set, get, or remove them. These are useful if admins want to store information that should not be leaked to any client.
• The key $$token cannot be removed even with admin credentials.

Configurations

NUN_ELECTION_TIMEOUT

• Configurations are available to define the timeout period for elections to wait until they are acknowledged from all nodes. It is important to note that you should rarely change this variable since doing so could make elections slower. The value of this variable should be at least twice the latency value to ensure that the election process runs smoothly.