Description: Setting up and configuring a multi-region CockroachDB cluster on Azure AKS Tags: Azure, AKS, Kubernetes, K8s, data
Edit the file env.sh, setting each of the variables to suit your application.
vm_type="Standard_E2d_v4"
n_nodes=3
rg="$USER-aks-multi-region"
loc1="westus"
loc2="centralus"
loc3="eastus"
clus1="crdb-aks-$loc1"
clus2="crdb-aks-$loc2"
clus3="crdb-aks-$loc3"
Run this script to create a resource group (RG) for the project.
In order to enable VPC peering between the regions, the CIDR blocks of the VPCs must not overlap. This value cannot change once the cluster has been created, so be sure that your IP ranges do not overlap.
Run this script to create the the virtual networks creation and peer them.
Run This script to create a Kubernetes (K8s) cluster in each of the three regions.
When prompted about whether or not to proceed with each process, type y
.
Run this script to configure kubectl.
You can switch K8s contexts using this approach:
kubectl config use-context crdb-aks-$clus1
where $clus1
, $clus2
and $clus3
are defined in your ./env.sh
file.
(optional) Run this script to test network connectivity.
Download it here and
install the cockroach
binary into a directory in your PATH
.
The cockroach
binary will be used to generate certificates.
If cockroach
is not on your PATH
, edit in the ./multiregion/setup.py
, assigning
the cockroach_path
variable to the path to the cockroach
binary.
Run this script to generate the context
and
regions
maps you'll embed into ./multiregion/setup.py
.
./06_contexts_regions.sh
# Replace the existing contexts and regions definitions in setup.py with these:
contexts = { 'westus': 'crdb-aks-westus', 'centralus': 'crdb-aks-centralus', 'eastus': 'crdb-aks-eastus' }
regions = { 'westus': 'westus', 'centralus': 'centralus', 'eastus': 'eastus' }
Use this output to edit setup.py
.
NOTE: the regions
map just maps each region to itself.
cd ./multiregion/
python setup.py
cd -
As the script creates various resources and creates and initializes the
CockroachDB cluster, you'll see a lot of output, eventually ending with job "cluster-init-secure" created
.
Each Kubernetes cluster has a CoreDNS service that responds to DNS requests for pods in its region. CoreDNS can also forward DNS requests to pods in other regions.
To enable traffic forwarding to CockroachDB pods in all 3 regions, you need to modify the ConfigMap for the CoreDNS Corefile in each region.
There are three sample config maps here.
- Run This script to create the three
configmap-*.yaml
files you need in the next step.
These files will be named configmap-
, the name of the region (the $loc*
values from env.sh
), then .yaml
.
- (optional) Back up the existing config maps (do this for each region):
kubectl -n kube-system get configmap coredns -o yaml > <configmap-backup-name>
-
Run this script to apply each of the three config maps.
-
For each region, check that your CoreDNS settings were applied:
kubectl get -n kube-system cm/coredns -o yaml --context <cluster-context>
- Confirm that the CockroachDB pods in each cluster report
1/1
in theREADY
column, indicating that they've successfully joined the cluster. NOTE: This could take from a couple of minutes to hours to take effect. Be patient.
. ./env.sh
kubectl get pods --selector app=cockroachdb --all-namespaces --context $clus1
NAMESPACE NAME READY STATUS RESTARTS AGE
us-east1-b cockroachdb-0 1/1 Running 0 14m
us-east1-b cockroachdb-1 1/1 Running 0 14m
us-east1-b cockroachdb-2 1/1 Running 0 14m
kubectl get pods --selector app=cockroachdb --all-namespaces --context $clus2
NAMESPACE NAME READY STATUS RESTARTS AGE
us-central1-a cockroachdb-0 1/1 Running 0 14m
us-central1-a cockroachdb-1 1/1 Running 0 14m
us-central1-a cockroachdb-2 1/1 Running 0 14m
kubectl get pods --selector app=cockroachdb --all-namespaces --context $clus3
NAMESPACE NAME READY STATUS RESTARTS AGE
us-west1-a cockroachdb-0 1/1 Running 0 14m
us-west1-a cockroachdb-1 1/1 Running 0 14m
us-west1-a cockroachdb-2 1/1 Running 0 14m
- SSH approach to get a SQL prompt:
kubectl exec --stdin --tty cockroachdb-0 --namespace $loc3 --context $clus3 -- /bin/bash
Then, start the SQL CLI:
# ./cockroach sql --certs-dir ./cockroach-certs
- Creation of secure client:
kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/client-secure.yaml --namespace $loc1
kubectl exec -it cockroachdb-client-secure -n $loc1 -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
NOTE: you must first create a user with admin rights so that you can log in to the console.
kubectl port-forward cockroachdb-0 8080 --context $clus1 --namespace $loc1
./09_teardown.sh