databricks/docker-dev

Arcion Demo Kit for testing database to database replication

More info at Arcion Demo Kit docs.

Overview

This is the Arcion Demo Kit. It is designed to demo and test Arcion replication from various data sources to targets. The diagram below describes the components of the demo kit. Please refer to https://docs.arcion.io for more info.

• Load Generators
  • Yahoo Cloud Serving Benchmark YCSB
  • Carnegie Mellon Database Group BenchBase
• Data sources
• Arcion cluster with dedicated metadata database
• Data destinations

graph LR
    L[Load Generator<br>TPC-C<br>YCSB] --> S
    subgraph Arcion Cluster
        A1
        M[(Meta <br>Data)]
    end
    S[(Source <br>Data)] --> A1[Arcion <br> UI]
    A1 --> T[(Destination<br>Data)]

Loading

Getting started

Assumptions:

• Run on Windows WSL2, Liunx or Mac
  • x64_64 (Intel, AMD) CPUs can run all databases
  • ARM64 (Apple Silicon, Tau, Graviton2) can run Oracle 19c
• Access to a terminal
• Access to a browser
• Arcion License file replicant.loc

Install and Setup

OSX (Mac) prerequisites

brew install dialog 
brew install jq 
brew install git
brew install wget
brew install bash
brew install podman 
brew install podman-desktop
pip3 install podman-compose

• bash required for Demokit's install.sh

echo $(brew --prefix)/bin/bash | sudo tee -a /private/etc/shells
chpass -s $(brew --prefix)/bin/bash

• podman with 512GB of disk, 16GB of RAM, and 8 CPUs

podman machine init --disk-size 512 --memory 16384 --cpus 8
podman machine start

Install Arcion Demo Kit

Cut and paste the following in a terminal.

git clone https://github.com/arcionlabs/docker-dev
cat >>~/.profile <<EOF
export ARCION_LICENSE="$(cat replicant.lic | base64)"
export DOCKERDEV_BASEDIR="$(pwd)/docker-dev"
EOF
source ~/.profile
docker-dev/install.sh

Podman example

Docker exmaple

Demo Recordings

Arcion supports snapshot, real-time, snapshot, and delta-snapshot replication modes. Recorded Arcion CLI demos of source, destination, and replication type combination is available. The recordings use asciinema so that the YAML config files can be cut/pasted. This is a functional demo using TPCC and YCSB. The data size is 1GB each, 1 thread given to Arcion, and 8 GB of RAM shared.

• The source is on the left side of the table (the left column).
• The destination is on the right side of the table (the top row).
• The cell has workloads and URL to the demo.
• A blank cell means a demo has not been recorded as of yet.

Snapshot Replication CLI Demos

snapshot	kafka	minio	mysql	null	oracle	pg	redis stream	snowflake
db2 luw	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB
informix	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB
mysql	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB
oracle	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB
pg	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB	TPCC, YCSB		TPCC, YCSB		TPCC, YCSB

Full Replication CLI Demos

full	databricks	Google Cloud Storage	Google CloudSQL MySQL	kafka	minio	mongodb	mysql	null	oracle	pg	redis stream	singlestore	snowflake	sqlserver
db2 luw							TPCC YCSB		TPCC YCSB	TPCC YCSB
IBM Db2									various
informix				YCSB	YCSB		YCSB	YCSB	YCSB,YCSB	YCSB,YCSB	YCSB		YCSB
mongodb
mysql	various	TPCC YCSB	TPCC YCSB	TPCC YCSB	TPCC YCSB		TPCC YCSB,TPCC & YCSB,TPCC & YCSB	TPCC YCSB	TPCC YCSB	TPCC YCSB	TPCC YCSB	various
oracle				TPCC YCSB,TPCC YCSB	TPCC YCSB	various,	TPCC YCSB	TPCC YCSB	various	TPCC YCSB		various	YCSB,
pg				YCSB	YCSB		YCSB		X	YCSB	YCSB			YCSB
snowflake														various

Using Demo Kit

CLI

The demo kit uses tmux. Based on your preference, use one or both methods below. Both views will be in sync.

• using a browser: http://localhost:7681
• using a terminal: docker exec -it workloads tmux attach

UI

Go to http://localhost:8080 and sign in with user admin password arcion

Running a demo with CLI

each will run for 5 minutes and times out by default

arcdemo.sh full mysql kafka
arcdemo.sh full mysql minio
arcdemo.sh full mysql pg

Change Scale Factor Performance and Scale Tests

• For 1GB volume test, change the scale factor to 10

  go to http://localhost:7681

  each will run for 5 minutes and times out by default

  scale factor 10 will generate about 1GB of data on YCSB and 1GB TPC-C

  arcdemo.sh -s 10 full mysql kafka
  arcdemo.sh -s 10 full mysql minio
  arcdemo.sh -s 10 full mysql pg

• For 10GB volume test, change the scale factor to 100

  go to http://localhost:7681

  each will run for 5 minutes and times out

  scale factor 100 will generate about 10GB of data on YCSB and 1GB of TPC-C set snapshot inter table parallelism to 2 on the extractor and 2 on the applier

  arcdemo.sh -s 100 -b 2:2 full mysql kafka
  arcdemo.sh -s 100 -b 2:2 full mysql pg
  arcdemo.sh -s 100 -b 2:2 full mysql minio

• For stress test of CDC, change the workload update rate and increase threads on Arcion real-time threads

  arcdemo.sh -s 100 -b 2:2 -c 2:2 -r 0 full mysql kafka
  arcdemo.sh -s 100 -b 2:2 -c 2:2 -r 0 full mysql pg
  arcdemo.sh -s 100 -b 2:2 -c 2:2 -r 0 full mysql minio

  -r 2:2 use 2 threads for Arcion real-time extractor and applier respectively -t 0 run YCSB on 1 thread and TPC-C on 1 thread as fast as possible

To shutdown all data source and destination providers

for db in $( find * -maxdepth 2 -name docker-compose.yaml ); do
pushd $(dirname $db) 2>/dev/null; docker compose stop; popd 2>/dev/null
done

• down removes the container and its running image

for db in $( find * -maxdepth 2 -name docker-compose.yaml ); do
pushd $(dirname $db) 2>/dev/null; docker compose down; popd 2>/dev/null
done

for db in $( find * -maxdepth 2 -name docker-compose.yaml ); do
pushd $(dirname $db); docker compose up -d; popd
done

Cloud Database Examples

Snowflake

• Snowflake source to MySQL destination use the default on mysql destination

single thread each extractor and applier source catalog is SNOWFLAKE_SAMPLE_DATA and source schema is TPCH_SF1

SRCDB_DB=SNOWFLAKE_SAMPLE_DATA SRCDB_SCHEMA=TPCH_SF1 arcdemo.sh snapshot snowflake mysql

two threads each extractor and applier source catalog is default arcsrc and source schema is PUBLIC

arcdemo.sh -b 2:2 snpashot snowflake mysql

Oracle Docker Setup

Oracle requires container images to be built locally. Start with Oracle XE, then use Oracle EE for volume testing. Oracle XE does not require the extra step of downloading the Oracle EE binary. Oracle EE should be used for anything scale factor beyond 10.

Oracle XE

• Build the image

  cd oracle
  git clone https://github.com/oracle/docker-images oracle-docker-images
  pushd oracle-docker-images/OracleDatabase/SingleInstance/dockerfiles 
  ./buildContainerImage.sh -v 21.3.0 -x -o '--build-arg SLIMMING=false'
  popd
  cd ..

• Start service

  docker compose -f oraxe2130/docker-compose.yaml up -d

• A test examples

  Scale factor 10 Snapshot inter table parallelism of 2

  arcdemo.sh -s 10 -b 2:2 full oraxe pg

Testing with different tags of demokit

• to install specific tag

export ARCION_WORKLOADS_TAG=23.07
/bin/bash -c "$(curl -k -fsSL https://raw.githubusercontent.com/arcionlabs/docker-dev/${ARCION_WORKLOADS_TAG:-HEAD}/install.sh)"

• to install DBs not listed in the menu

export ARCION_DOCKER_DBS=(db2 informix)
/bin/bash -c "$(curl -k -fsSL https://raw.githubusercontent.com/arcionlabs/docker-dev/${ARCION_WORKLOADS_TAG:-HEAD}/install.sh)"

Generate Source / Target Matric

cd bin
./startall.sh
./recdemo.sh