- kill clusters
- remove clusters from Anthos
- make sure disks are cleaned up here: Google Cloud Platform
<spin up, then we chat> What’s Anthos?
- manage multiple k8 clusters as fleets
- config management -> git ops + k8s policy management
- service mesh -> IP visibility and security
- multi and hybrid cloud
What’s Cassandra?
- Distributed database designed for availability and scale (in CAP terms, C* is AP, while most distributed db’s are CP)
- the data fabric behind huge distributed systems from Apple & Netflix to FedEx and Verizon
Why are they great together?
- Anthos makes it more efficient than ever to scale your applications across the world for great uptime and responsiveness. Cassandra is an idea database to support modern, globally-distributed app (big or small).
Fleets!
Where? Anywhere… we will use GCP but this could be on-premises or even on another cloud. Anthos provides us the tools to manage hybrid and multi cloud deployments with great visibility, security, and repeatability
To set our terms: Hybrid: on-premises -> cloud Multi: cloud -> cloud
Lets spin up a Kubernetes cluster: — in production we would actually split these nodes across zones in the region, Cassandra can use that when placing replicas to make the data even more durable
5 minutes
gcloud beta container \
--project "anthos-309114" \
clusters create "cen-cassandra" \
--zone "us-central1-c" \
--no-enable-ip-alias \
--machine-type "e2-standard-4" \
--num-nodes "3" \
--network "projects/anthos-309114/global/networks/default" \
--subnetwork "projects/anthos-309114/regions/us-central1/subnetworks/default" \
--node-locations "us-central1-c" \
--addons GcePersistentDiskCsiDriver
Or single:
gcloud beta container \
--project "anthos-309114" \
clusters create "cen-cassandra" \
--zone "us-central1-c" \
--no-enable-ip-alias \
--machine-type "e2-standard-4" \
--num-nodes "1" \
--network "projects/anthos-309114/global/networks/default" \
--subnetwork "projects/anthos-309114/regions/us-central1/subnetworks/default" \
--node-locations "us-central1-c" \
--addons GcePersistentDiskCsiDriver
Define both…
Cassandra is a NoSQL database designed for distributed deployment (think NY, LA and Tokyo). It’s great for problems that need super high availability (no leaders, configured correctly you can loose a node, an AZ or even and entire DC/region w/o interuption)
K8ssandra is an Kubernetes operator plus a set of useful tools for daily life with Cassandra:
- medusa -> tools to manage Cassandra backups
- reaper -> long running anti-entropy data repair (not the first line of defense against data entropy in Cassandra)
- grafana -> dashboard for viewing cluster metrics, though you may want to look at integrating directly with GCP metrics tools.
Cluster is done? Let’s grab our creds for our first GKE cluster
gcloud container clusters get-credentials cen-cassandra --region us-central1-c --project anthos-309114
In k8ssandra we can setup these credential with Kubernetes secrets.
kubectl create namespace us-central1
kubectl create secret generic cassandra-admin-secret \
--from-literal=username=cassandra-admin \
--from-literal=password=cassandra-admin-password \
-n us-central1
If you try this at home, please don’t use this username and password!!! (And go read the Cukoo’s Egg, if you haven’t yet)
Now to setup our config for the first K8ssandra cluster. Check out our gke.yaml
vim gke.yaml
Call out
- auth settings, and how they tie to the secret we created
- cluster name, and how it will need to match other k8ssandra installs for replication
- size, and cluster size… might be good to share a version of the config with multiple dc ’s and racks… talk about Cassandra being smart about striping the data. -> “here we are talking about keeping data close to users, but lots of C* users care about 100% uptime or greater r/w speeds and do that by adding nodes in each DC and striping them across AZ’s. Loose a node, loose and AZ, loose a region…”
- promethious/grafana
- Cassandra version
- data volume size (and stateful sets)
- number of nodes, 3 is most common (best avialabilty both in region and distributed, consistent latency metrics). 1 is also possible, but loosing that node would take out an entire regions.
(You could also import your helm into Anthos config management, but we aren’t there yet…) 5 minutes
helm repo add k8ssandra https://helm.k8ssandra.io
helm repo update
helm install anthos-k8ssandra k8ssandra/k8ssandra -f k8ssandra/gke.triple.cen.yml -n us-central1
Did it go?
watch kubectl get pods -n us-central1
Problems?
kubectl -n us-central1 get events
kubectl -n us-central1 exec \
-it multi-region-dc2-default-sts-0 \
-c cassandra \
-- nodetool -u cassandra-admin -pw cassandra-admin-password status
- reaper
- cass-operator
- grafana
- prometheus
- stargate
Wait for the Stargate pod and we can proceed
kubectl exec -it multi-region-dc2-default-sts-0 \
-c cassandra \
-n us-central1 \
-- nodetool -u cassandra-admin -pw cassandra-admin-password status
UN
is up and normal, our C* DC is ready to go!
In ui, register cluster to fleet Google Cloud Platform
Url: https://github.com/omnifroodle/lunch.git Authentication: none Branch: us-central1 Policy directory: Anthos
show config management directory, call out password secret
TODO sneak in second cluster creation!
New namespace? Also needs a secret
kubectl create namespace frontend
kubectl create secret generic cassandra-admin-secret --from-literal=username=cassandra-admin --from-literal=password=cassandra-admin-password -n frontend
kubectl apply -f lunch-deployment.yaml -n frontend
kubectl expose deployment lunch-app --name=lunch-app-service --type=LoadBalancer --port 80 --target-port 4000 -n frontend
Watch for an external ip here:
❯ kubectl -n frontend get services ─╯
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
anthos-k8ssandra-dc2-stargate-service ClusterIP 10.112.4.177 <none> 8080/TCP,8081/TCP,8082/TCP,8084/TCP,8085/TCP,9042/TCP 95m
anthos-k8ssandra-grafana ClusterIP 10.112.2.251 <none> 80/TCP 95m
anthos-k8ssandra-kube-prom-operator ClusterIP 10.112.6.188 <none> 443/TCP 95m
anthos-k8ssandra-kube-prom-prometheus ClusterIP 10.112.3.101 <none> 9090/TCP 95m
anthos-k8ssandra-reaper-reaper-service ClusterIP 10.112.0.216 <none> 8080/TCP 95m
cass-operator-metrics ClusterIP 10.112.2.95 <none> 8383/TCP,8686/TCP 95m
lunch-app-service LoadBalancer 10.112.12.109 35.239.42.95 80:31020/TCP 37m
multi-region-dc2-all-pods-service ClusterIP None <none> 9042/TCP,8080/TCP,9103/TCP 95m
multi-region-dc2-service ClusterIP None <none> 9042/TCP,9142/TCP,8080/TCP,9103/TCP,9160/TCP 95m
multi-region-seed-service ClusterIP None <none> <none> 95m
prometheus-operated ClusterIP None <none> 9090/TCP 95m
kubectl -n us-central1 exec -it multi-region-dc2-default-sts-0 -c cassandra -- cqlsh -u cassandra-admin -p cassandra-admin-password
single
create KEYSPACE lunch WITH replication = {'class' : 'NetworkTopologyStrategy', 'dc2' : 1};
create table lunch.lunchers ( location text, name text, time timestamp, thumbnail text, primary key(location, time, name)) WITH clustering order by (time desc);
kubectl -n frontend rollout restart deployment/lunch-app
?web ui?
Talk about it, probably not worth the time to roll out
Security Observability
Anthos service mesh… slow to add (maybe do this right after cluster creation?!)
./install_asm \
--project_id anthos-309114 \
--cluster_name cen-cassandra \
--cluster_location us-central1-c \
--mode install \
--output_dir ./asm-downloads \
--enable_all
TODO this can take a while, maybe Rohit can talk about Anthos more here?
kubectl -n istio-system get pods -l app=istiod --show-labels
kubectl label namespace us-central1 istio-injection- istio.io/rev=asm-1102-3 --overwrite
Let’s add some more cloud, apps should be everywhere… data fabric… repeatable parts… SRE, etc
To build a cluster that replicates across regions (or clouds, or on-prem) we’ll need to make sure we have a couple things in common across our k8ssandra installs:
- cluster seeds, more on this later
- admin credentials
- a shared cassandra cluster name
- network access between nodes (and K8s clusters)
First, we need seeds to ensure we know how to connect our clusters…
Get seeds:
kubectl get pods -n us-central1 -o jsonpath="{.items[*].status.podIP}" --selector cassandra.datastax.com/seed-node=true
10.0.0.18
Set additional seeds in gke.triple.east.yml
We’ll need another cluster:
gcloud beta container \
--project "anthos-309114" \
clusters create "east-cassandra" \
--zone "us-east1-b" \
--no-enable-ip-alias \
--machine-type "e2-standard-4" \
--num-nodes "1" \
--network "projects/anthos-309114/global/networks/default" \
--subnetwork "projects/anthos-309114/regions/us-east1/subnetworks/default" \
--node-locations "us-east1-b" \
--addons GcePersistentDiskCsiDriver
Let’s grab our creds for our new GKE cluster
gcloud container clusters get-credentials east-cassandra --region us-east1-b --project anthos-309114
Add our secrets:
kubectl create namespace us-east1
kubectl create secret generic cassandra-admin-secret --from-literal=username=cassandra-admin --from-literal=password=cassandra-admin-password -n us-east1
And our helm chart:
helm install anthos-k8ssandra k8ssandra/k8ssandra -f k8ssandra/gke.single.east.yml -n us-east1
When we think pods are done:
kubectl -n us-east1 exec -it multi-region-dc1-default-sts-0 \
-c cassandra \
-- nodetool -u cassandra-admin -pw cassandra-admin-password status
Check our firewall
TIME HERE TO DISCUSS NETWORKING OPTIONS for multi/hybrid (adding nodes will take a few minutes)
Config manager here
kubectl -n us-east1 exec -it multi-region-dc1-default-sts-0 -c cassandra -- cqlsh -u cassandra-admin -p cassandra-admin-password
Switch to a us-east1 branch
drop keyspace lunch;
ALTER KEYSPACE system_auth WITH REPLICATION =
{'class' : 'NetworkTopologyStrategy', 'dc2' : 1, 'dc1': 1};
Cassandra and Anthos are complimentary technologies that work together to make building distributed apps easier, as well as making operating them safer and faster.
Whatever your reason for a distributed architecture, be it hybrid cloud as a way to access burstable cloud resource (or a way to begin the cloud life of on-prem assets), apps that are responsive around the world (no waiting on round trips to a database instance on the other side of the world), or apps that just can’t go down…
Anthos and Cassandra are a great choice to make them happen!
Please, reach out to us at TK