Distributed Cassandra in 2 Kubernetes Clusters

These scripts deploy a Cassandra ring across 2 kubernetes clusters.

The 2 k8s clusters are provisioned into Azure using ARM templates generated by acs-engine.
Each cluster runs in its own Resource Group and Virtual Network. Networks are not overlapping to allow for VNET connections.
Clusters are configured to manage container IP addresses using CNI, to have one IP address space for hosts and containers.
Virtual Networks are connected via Virtual Network Gateways to allow for cross-region, cross-site or cross-cloud deployments or Global VNET peering.
Each cluster configures a GossipingPropertyFileSnitch with information about the node from the Azure Metadata Service

Branches

master builds acs clusters configured for Azure CNI connected via VNET Gateways
peering builds acs clusters configured for Azure CNI connected via Global VNET Peering

How it works

The script works in 2 phases.

The first provisions 2 k8s clusters in 2 Azure regions with network connectivity betweent the 2 clusters.

The second phase provisions a Cassandra ring across the 2 clusters from the helm charts built by Microsoft's Partner Catalyst Team.

The second phase could easily be adapted to provision other hybird applications into the cluster.

Cluster Provisioning

The script first creates ARM templates for the 2 clusters by running acs-engine. Each template contains cluster specific SSL certificates tied to the DNS Prefix. The API model for the cluster is based on the VNET example. The cluster is configured for CNI. CNI configuration avoids complexities of managing the ACS route table across networks.

For configuring k8s clusters connected in clusters with ACS networking defaults, see my setup guide for connected k8s clusters.

Next the script provisions 2 Resource Groups, each with a VNET, Subnets and the VNET Gateway. Address spaces, DNS Prefixes, etc. are customizable via variables in the script. Provisioning the gateways is asynchrnous since it can take up to 45 minutes.

The deployment proceeds with provisioning the k8s clusters into the 2 Resource Groups.

Once the clusters are provisioned, the script will wait for Gateway provisioning to complete. Once both gateways are finished, the script will provisiong Cassandra.

Cassandra Provisioning and Configuration

The script installs cassandra using helm charts. The Cassandra deployment is Azure Fault Domain aware. Be sure to read up on implications

The first deployment is a standard Cassandra cluster, with the Seed node pointing to a Cassandra service configured to front all available containers.

The second cluster configures a Seed node in the first cluster. Currently, the first container in the first cluster is chosen as the Seed node.

After provisioning of the 2nd cluster completes, the Cassandra ring is complete. Run nodetool status in on of the containers to verify. You'll see nodes in 2 data centers:

$ kubectl exec -it $(kubectl get pods -o jsonpath='{ .items[0].metadata.name }') /usr/local/apache-cassandra-3.11.0/bin/nodetool status
Datacenter: dc-eastus
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.1.0.101  99.62 KiB  32           31.9%             4e3f087d-62b5-4def-a8db-59fc18e4c023  rack-1
UN  10.1.0.23   99.62 KiB  32           30.1%             15559b3a-ce06-4898-a952-8464ff884204  rack-0
UN  10.1.0.59   99.62 KiB  32           38.6%             5a03882d-cbea-4cdc-aefa-f75e98d8f728  rack-2
Datacenter: dc-southcentralus
=============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.2.0.50   99.62 KiB  32           31.1%             1fde7c7b-dc7d-4344-9e71-d27d66313dd4  rack-0
UN  10.2.0.112  99.62 KiB  32           32.6%             e45c777f-3805-49cd-a13d-7a0da8c80a44  rack-1
UN  10.2.0.93   99.62 KiB  32           35.7%             117dad13-ebe4-49f3-9309-20aa35427f79  rack-2

Running the Script

Execute the script by running

./deploy-multidc.sh [resourcegroupname1] [resourcegroupname2]

You will need acs-engine available in your environment.

Variables

CLUSTER_DEFINITION_x: API Model for Cluster x
VNET_NAME: Name for Cluster VNet
SUBNET_NAME: Name for Cluster VNet
VNET_x_FIRST_TWO: First 2 octets of the Vnet x address range
LOCATION_x: Deployment Location for Resource Group x
SERVICE_PRINCIPAL: Service Principal Client ID used by k8s. Needs at least Contributor Access to the RG
SP_SECRET: Service Principal Secret (password)
DNS_PREFIX_x: DNS prefix for cluster x
SSH_PUBLIC_KEY: Public Key for admin account.

Networks

The script is setting up 2 clusters with non-overlapping address spaces"

Cluster 1

VNet: 10.1.0.0/16
Agent/Container Subnet: 10.1.0.0/17
GatewaySubnet: 10.1.128.0/29

Cluster 2

VNet: 10.2.0.0/16
Agent/Container Subnet: 10.2.0.0/17
GatewaySubnet: 10.2.128.0/29

acs-engine API model

    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "kubernetesConfig": {
        "networkPolicy": "azure"
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "",
      "vmSize": "Standard_D2_v2",
      "firstConsecutiveStaticIP": "10.1.127.250"    
    },

Verifying Replication

Ssh into a container kubectl exec -it $(kubectl get pods -o jsonpath='{ .items[0].metadata.name }') bash
install pip apt-get intall python-pip
(if you get an error like ImportError: No module named cqlshlib, then install the cassandra driver pip install cassandra-driver and set the environment variable export CQLSH_NO_BUNDLED=true, exit container, log back on)
install cqlsh pip install cqlsh
Connect Cassandra cqlsh -u cassandra -p cassandra --cqlversion="3.4.4"
Create 2 records

CREATE KEYSPACE customers_ks WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'dc-eastus' : 3, 'dc-southcentralus' : 3}; 

USE customers_ks;

CREATE TABLE Customers(customer_id int PRIMARY KEY, firstname text, lastname text); 

INSERT INTO Customers(customer_id, firstname, lastname) VALUES(1, 'John', 'Doe'); 
INSERT INTO Customers(customer_id, firstname, lastname) VALUES (2, 'Jane', 'Doe');

Verify records exist: SELECT * FROM Customers; Expected Output:

SELECT * FROM Customers;

 customer_id | firstname | lastname
-------------+-----------+----------
           1 |      John |      Doe
           2 |      Jane |      Doe

(2 rows)

---

Log on to container in the other region and get cqlsh ready.
USE customers_ks;
Query Customers as above SELECT * FROM Customers;
Note same output as above.

Other Things That Could be Done

Improve Seed Node selection
Port to DC/OS
Optimize SSH calls in the script
Fix Cassandra chart agent node selection, to not use Azure VM Skus
Investigate Kubernetes federation as deployment mechanism, similar to my HA configuration with CosmosDb.
Cross Cloud Setup
Install VNET, GW and Cluster with single ARM template

xtophs/acs-k8s-cassandra-multi-dc