What This Repo is For

This repo captures instructions for setting up a cluster_run based build for the HA demo for Connect SV 2018.

Setup

First clone this repo and sync and build 5.5.1 Couchbase Server:

git clone git@github.com:dave-finlay/ha-demo-connect-2018.git
repo init -u git://github.com/couchbase/manifest -m released/5.5.1.xml -g all
repo sync
make -j4 EXTRA_CMAKE_OPTIONS='-DCOUCHBASE_DISABLED_UNIT_TESTS="kv_engine;platform"'

Change to the demo directory:

cd ha-demo-connect-2018

Edit your .bash_login so that new terminal windows automatically cd to ha-demo-connect-2018 directory.

Start the cluster. Best to do so in 6 separate terminals:

start-server.sh 0
start-server.sh 1
start-server.sh 2
start-server.sh 3
start-server.sh 4
start-server.sh 5

Create the cluster. This will set up the server groups, add the nodes, create the bucket, make sure the auto-failover settings are correct and rebalance:

create-cluster.sh

Open a couple of browser tabs and place them side-by-side:

open-browser-tabs.sh

Start the workload:

start-workload.sh

Demo Steps

0. Introduce setup

6 node cluster
1 bucket - the messages bucket
2 replicas
3 server groups

1. Create document

cat post-important-message.sh
post-important-message.sh
open-buckets.sh # and navigate Documents \ Classic Editor \ important-message
show-message-vbucket.sh
show-message-vbucket.sh 1000

2. Single node failure.

Drop server 2.
Return to terminal and see the vbmap change
Observe effect on workload, before and after failover.
open-message.sh # to show message still around after failover

3. Another node fails.

Open another terminal window
change-mode-on-node.sh 5 -w
Kill this terminal window and show the terminal with show-message-vbucket.sh running.
Observe effect on workload, before and after failover.

4. Recover cluster.

Navigate to server 2 terminal and restart server 2
Open terminal window
change-mode-on-node.sh 5 +w
Kill terminal window
Add back servers 2 and 5 via delta-node recovery.
Rebalance

5. Orchestrator failure.

find-orchestrator.sh # to show which node is the orchestrator
Hang the orchestrator:

send-signal-to-node.sh 0 STOP

Observe failover and workload.
Switch to logs page.
Unhang the orchestrator.

send-signal-to-node.sh 0 CONT

Add back. Rebalance.

6. Drop server group 3.

Observe failover and workload.

Handy Commands You Might Need

List server groups:

../couchbase-cli/couchbase-cli group-manage -c 127.0.0.1:9000 -u Administrator -p asdasd  --list

Delete a server group:

../couchbase-cli/couchbase-cli group-manage -c 127.0.0.1:9000 -u Administrator -p asdasd  --group-name "Group 5" --delete

Show important-message on the command line:

curl -s localhost:9000/pools/default/buckets/messages/docs/important-message -u Administrator:asdasd | jq -r .json