/junodb

JunoDB is PayPal's home-grown secure, consistent and highly available key-value store providing low, single digit millisecond, latency at any scale.

Primary LanguageGoApache License 2.0Apache-2.0

JunoDB - A secure, consistent and highly available key-value store

License Build Docker

What is JunoDB

JunoDB is PayPal's home-grown secure, consistent and highly available key-value store providing low, single digit millisecond, latency at any scale.

Getting Started with the JunoDB Server

Clone the repository from github

git clone https://github.com/paypal/junodb.git

Set BUILDTOP variable

export BUILDTOP=<path_to_junodb_folder>/junodb
cd $BUILDTOP

Continue building JunoDB server with

  1. Docker build or
  2. Manual build

Docker Build

Install Dependencies

Install Docker Engine version 20.10.0+ Check for existing docker version

docker version

Install Docker if not installed or version is older than 20.10.0

docker/setup.sh
#If you are not added to the docker group, you will have to logout and login in the machine after running docker/setup.sh

If the user is not added to docker group, you may add manually. Logout and re-login after this step.

sudo usermod -a -G docker $USER

Verify that your username is added to docker group

groups

#or
cat /etc/group | grep docker

Build JunoDB

#Login to docker hub account 
docker login

# Build junodb docker images
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient


docker/build.sh 

Run JunoDB

# Setup junodb network and start junodb services
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient


# JunoDB proxy service listens on port 
# :5080 TLS and :8080 TCP
docker/start.sh 

Shutdown JunoDB services

# This will shutdown junodb services
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient

docker/shutdown.sh 

Manually Run JunoDB services

#This can be done instead of ./start.sh to start up the docker services

cd $BUILDTOP/docker/manifest

# To run junodb services in --detach mode (recommended)
docker compose up -d

# Juno proxy service listens on port 
# :5080 TLS and :8080 TCP

#To view the running containers 
docker ps

# To stop junodb services
docker compose down

Generate Secrets for Dev


NOTE: secrets for TLS and Encryption can be generated for dev/testing.

sh $BUILDTOP/docker/manifest/config/secrets/gensecrets.sh

## generated secrets
# server.crt/server.pem - certificate/key for junodb proxy for TLS 
# ca.crt - CA cert
# keystore.toml - sample keystore file

Validate JunoDB

Login to docker client and check connection with proxy

docker exec -it junoclient bash -c 'nc -vz proxy 5080'

You can also test the junodb server by running junocli and junoload


JunoCLI

The following commands log in to the docker client and run the ./junocli command directly. The proxy ip is aliased as "proxy"

  1. CREATE
docker exec -it junoclient bash -c '/opt/juno/junocli create -s proxy:8080 -c config.toml -ns test_ns test_key test_value'
  1. GET
docker exec -it junoclient bash -c '/opt/juno/junocli get -s proxy:8080 -c config.toml -ns test_ns test_key'
  1. UPDATE
docker exec -it junoclient bash -c '/opt/juno/junocli update -s proxy:8080 -c config.toml -ns test_ns test_key test_value_updated'
  1. DESTROY
docker exec -it junoclient bash -c '/opt/juno/junocli destroy -s proxy:8080 -c config.toml -ns test_ns test_key'

More about junocli here


Junoload

The following command logs in to the docker client and runs the ./junoload command directly. Junoload can be used for benchmarking juno server. The proxy ip is aliased as "proxy"

docker exec -it junoclient bash -c '/opt/juno/junoload -s proxy:5080 -ssl -c config.toml -o 1'

More about junoload here



Manual Build

The following sections explain the process for manually building the JunoDB server without Docker. These instructions are based on an Ubuntu 20.04.5 system

Install Dependencies

Install OpenSSL 1.0.2g+

sudo apt install openssl

Install multilog

sudo apt install daemontools

Install dependencies for rocksdb

sudo apt-get install build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev -y

Install Python
#install python
sudo apt-get install python3.8
#set soft link
cd /usr/bin
sudo ln -s python3.8 python

Build JunoDB

binary_build/build.sh

Run JunoDB

export JUNO_BUILD_DIR=$BUILDTOP/release-binary/code-build
script/deploy.sh

Validate JunoDB

#Validate if deploy was successful by checking if the proxy (junoserv), storage (junostorageserv), and etcd (junoclusterserv) processes are running
ps -eaf | grep juno
#There should be 41 processes running
#5 for junoclusterserv (3 logs, 1 etcdsvr.py, 1 etcdsvr_exe)
#20 for junostorageserv (6 logs, 1 manager, 12 workers, 1 monitor)
#16 for junoserv (6 logs, 1 manager, 8 workers, 1 monitor)

Test out the server using junocli and junoload command

See instructions for junocli here
See instructions for junoload here

Run functional tests

#Assuming user is in $BUILDTOP folder
test/functest/configsetup.sh
cd test/functest
$BUILDTOP/release-binary/tool/go/bin/go test -v -config=config.toml

Run unit tests

#Assuming user is in $BUILDTOP folder
cd test/unittest
$BUILDTOP/release-binary/tool/go/bin/go test -v

Shutdown JunoDB Services

#Assuming user is in $BUILDTOP folder
script/deploy.sh stop