Kubernetes Federated Cluster on AWS

Table of Contents

Create clusters
- Earth cluster
- Mars cluster
Setup federation
Deploy application
- Deploy Replica Set
- Deploy Service
Cleanup
AWS Commands

This repo shows how to create a federated Kubernetes cluster on Amazon Web Services using kops. Two clusters, each with 3 master and 3 worker nodes, are created in us-east-1 and us-east-2. One of the cluster is called earth and the other one is called mars. The federation is called interstellar.

The clusters are created on AWS using kops.

Create clusters

Create two clusters called earth and mars.

Earth cluster

Create hosted zone:

ID=$(uuidgen) && \
   aws route53 create-hosted-zone \
   --name earth.kubernetes-aws.io \
   --caller-reference $ID \
   | jq .DelegationSet.NameServers

Setup NS records in DNS provider. Make sure dig earth.kubernetes-aws.io NS return NS records:

dig earth.kubernetes-aws.io NS

; <<>> DiG 9.8.3-P1 <<>> earth.kubernetes-aws.io NS
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16919
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;earth.kubernetes-aws.io.	IN	NS

;; ANSWER SECTION:
earth.kubernetes-aws.io. 3271	IN	NS	ns-752.awsdns-30.net.
earth.kubernetes-aws.io. 3271	IN	NS	ns-1196.awsdns-21.org.
earth.kubernetes-aws.io. 3271	IN	NS	ns-1799.awsdns-32.co.uk.
earth.kubernetes-aws.io. 3271	IN	NS	ns-436.awsdns-54.com.

;; ADDITIONAL SECTION:
ns-436.awsdns-54.com.	162638	IN	A	205.251.193.180
ns-752.awsdns-30.net.	161105	IN	A	205.251.194.240
ns-1196.awsdns-21.org.	161034	IN	A	205.251.196.172
ns-1799.awsdns-32.co.uk. 148088	IN	A	205.251.199.7

;; Query time: 13 msec
;; SERVER: 10.4.4.10#53(10.4.4.10)
;; WHEN: Fri Aug 25 14:44:22 2017
;; MSG SIZE  rcvd: 245

Create cluster:

kops create cluster \
  --name=earth.kubernetes-aws.io \
  --master-count=3 \
  --master-zones us-east-1a,us-east-1b,us-east-1c \
  --node-count=3 \
  --zones=us-east-1a,us-east-1b,us-east-1c \
  --authorization=rbac \
  --state=s3://kubernetes-aws-io \
  --kubernetes-version=1.7.0 \
  --yes

Validate cluster:

kops validate cluster earth.kubernetes-aws.io --state=s3://kubernetes-aws-io
Validating cluster earth.kubernetes-aws.io

INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-1a	Master	m3.medium	1	1	us-east-1a
master-us-east-1b	Master	m3.medium	1	1	us-east-1b
master-us-east-1c	Master	c4.large	1	1	us-east-1c
nodes			Node	t2.medium	3	3	us-east-1a,us-east-1b,us-east-1c

NODE STATUS
NAME				ROLE	READY
ip-172-20-117-87.ec2.internal	node	True
ip-172-20-125-87.ec2.internal	master	True
ip-172-20-45-153.ec2.internal	master	True
ip-172-20-55-15.ec2.internal	node	True
ip-172-20-64-27.ec2.internal	master	True
ip-172-20-76-15.ec2.internal	node	True

Your cluster earth.kubernetes-aws.io is ready

Set alias for context:

kubectl config set-context \
    earth \
    --cluster=earth.kubernetes-aws.io \
    --user=earth.kubernetes-aws.io
Context "earth" created.

Get nodes:

kubectl --context=earth get nodes
NAME                            STATUS    AGE       VERSION
ip-172-20-117-87.ec2.internal   Ready     6m        v1.7.0
ip-172-20-125-87.ec2.internal   Ready     7m        v1.7.0
ip-172-20-45-153.ec2.internal   Ready     8m        v1.7.0
ip-172-20-55-15.ec2.internal    Ready     6m        v1.7.0
ip-172-20-64-27.ec2.internal    Ready     7m        v1.7.0
ip-172-20-76-15.ec2.internal    Ready     6m        v1.7.0

Mars cluster

Create hosted zone:

ID=$(uuidgen) && \
   aws route53 create-hosted-zone \
   --name mars.kubernetes-aws.io \
   --caller-reference $ID \
   | jq .DelegationSet.NameServers

Setup NS records in DNS provider. Make sure dig mars.kubernetes-aws.io NS return NS records:

dig mars.kubernetes-aws.io NS

; <<>> DiG 9.8.3-P1 <<>> mars.kubernetes-aws.io NS
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64774
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;mars.kubernetes-aws.io.		IN	NS

;; ANSWER SECTION:
mars.kubernetes-aws.io.	2630	IN	NS	ns-26.awsdns-03.com.
mars.kubernetes-aws.io.	2630	IN	NS	ns-964.awsdns-56.net.
mars.kubernetes-aws.io.	2630	IN	NS	ns-1052.awsdns-03.org.
mars.kubernetes-aws.io.	2630	IN	NS	ns-1965.awsdns-53.co.uk.

;; ADDITIONAL SECTION:
ns-26.awsdns-03.com.	143078	IN	A	205.251.192.26
ns-964.awsdns-56.net.	157684	IN	A	205.251.195.196
ns-1052.awsdns-03.org.	23776	IN	A	205.251.196.28
ns-1965.awsdns-53.co.uk. 75558	IN	A	205.251.199.173

;; Query time: 15 msec
;; SERVER: 10.4.4.10#53(10.4.4.10)
;; WHEN: Fri Aug 25 14:44:43 2017
;; MSG SIZE  rcvd: 243

Create cluster

kops create cluster \
  --name=mars.kubernetes-aws.io \
  --master-count=3 \
  --master-zones us-east-2a,us-east-2b,us-east-2c \
  --node-count=3 \
  --zones=us-east-2a,us-east-2b,us-east-2c \
  --authorization=rbac \
  --state=s3://kubernetes-aws-io \
  --kubernetes-version=1.7.0 \
  --yes

Validate

kops validate cluster mars.kubernetes-aws.io --state=s3://kubernetes-aws-io
Validating cluster mars.kubernetes-aws.io

INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-east-2a	Master	c4.large	1	1	us-east-2a
master-us-east-2b	Master	c4.large	1	1	us-east-2b
master-us-east-2c	Master	c4.large	1	1	us-east-2c
nodes			Node	t2.medium	3	3	us-east-2a,us-east-2b,us-east-2c

NODE STATUS
NAME						ROLE	READY
ip-172-20-107-105.us-east-2.compute.internal	node	True
ip-172-20-126-49.us-east-2.compute.internal	master	True
ip-172-20-41-181.us-east-2.compute.internal	node	True
ip-172-20-62-64.us-east-2.compute.internal	master	True
ip-172-20-89-187.us-east-2.compute.internal	node	True
ip-172-20-89-96.us-east-2.compute.internal	master	True

Your cluster mars.kubernetes-aws.io is ready

Set alias for context:

kubectl config set-context \
    mars \
    --cluster=mars.kubernetes-aws.io \
    --user=mars.kubernetes-aws.io
Context "mars" modified.

Get nodes:

kubectl --context=mars get nodes
NAME                                           STATUS    AGE       VERSION
ip-172-20-107-105.us-east-2.compute.internal   Ready     9m        v1.7.0
ip-172-20-126-49.us-east-2.compute.internal    Ready     10m       v1.7.0
ip-172-20-41-181.us-east-2.compute.internal    Ready     9m        v1.7.0
ip-172-20-62-64.us-east-2.compute.internal     Ready     10m       v1.7.0
ip-172-20-89-187.us-east-2.compute.internal    Ready     9m        v1.7.0
ip-172-20-89-96.us-east-2.compute.internal     Ready     10m       v1.7.0

Setup federation

Download k8s client binary:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/kubernetes-client-darwin-amd64.tar.gz
tar xzvf kubernetes-client-darwin-amd64.tar.gz

Check context:

kubectl config get-contexts
CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
          earth.kubernetes-aws.io   earth.kubernetes-aws.io   earth.kubernetes-aws.io
          mars                      mars.kubernetes-aws.io    mars.kubernetes-aws.io
*         mars.kubernetes-aws.io    mars.kubernetes-aws.io    mars.kubernetes-aws.io
          earth                     earth.kubernetes-aws.io   earth.kubernetes-aws.io

Use earth as host cluster:

kubectl config use-context earth
Switched to context "earth".

Create RBAC role binding:

kubectl create \
   clusterrolebinding \
   admin-to-cluster-admin-binding \
   --clusterrole=cluster-admin \
   --user=admin

Deploy the federation control plane in host cluster:

kubefed \
    init \
    interstellar \
    --host-cluster-context=earth \
    --dns-provider=aws-route53 \
    --dns-zone-name=kubernetes-aws.io

Shows the output:

Creating a namespace federation-system for federation system components... done
Creating federation control plane service..... done
Creating federation control plane objects (credentials, persistent volume claim)... done
Creating federation component deployments... done
Updating kubeconfig... done
Waiting for federation control plane to come up ....................................................................................... done
Federation API server is running at: a4f0f559a8dc111e786ba0a7088604ba-1703570671.us-east-1.elb.amazonaws.com

Get contexts again to see the newly created namespace:

kubectl config get-contexts
CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
          mars                      mars.kubernetes-aws.io    mars.kubernetes-aws.io
          mars.kubernetes-aws.io    mars.kubernetes-aws.io    mars.kubernetes-aws.io
*         earth                     earth.kubernetes-aws.io   earth.kubernetes-aws.io
          earth.kubernetes-aws.io   earth.kubernetes-aws.io   earth.kubernetes-aws.io
          interstellar              interstellar              interstellar

Change the context to federation context:
```
kubectl config use-context interstellar
```

Join earth and mars cluster to the federation:

kubefed join \
   earth \
   --host-cluster-context=earth \
   --cluster-context=earth
kubefed join \
   mars \
   --host-cluster-context=earth \
   --cluster-context=mars

--cluster-context defaults to cluster name

Check status of the clusters in the federation:

kubectl --context=interstellar get clusters
NAME      STATUS    AGE
earth     Ready     4h
mars      Ready     3h

Possible bug: kubernetes/kubernetes#51578

Create default namespace:
```
kubectl --context=interstellar create namespace default
```
This is required due to kubernetes/kubernetes#33292.

Deploy application

Deploy Replica Set

Deploy the Replica Set:

kubectl --context=interstellar create -f rs.yml

Verify that 3 pods are created in each cluster:
```
kubectl --context=interstellar get -w rs
NAME       DESIRED   CURRENT   READY     AGE
frontend   6         0         0         9m
```
Pods are not getting created even after 9 minutes. Filed kubernetes/kubernetes#51591.

Deploy Service

Deploy the Service:

kubectl --context=interstellar create -f svc.yml

Verify that 3 pods are created in each cluster:
```
kubectl --context=interstellar get -w rs
NAME       DESIRED   CURRENT   READY     AGE
frontend   6         0         0         9m
```
Similar error as above.

In addition, verify that ELB is created in two different regions and an external IP address is assigned.

Cleanup

Unjoin earth and mars cluster from the federation:

kubefed unjoin earth --cluster-context=earth
kubefed unjoin mars --cluster-context=earth

Delete clusters

kops delete cluster --name=earth.kubernetes-aws.io --state=s3://kubernetes-aws-io --yes
kops delete cluster --name=mars.kubernetes-aws.io --state=s3://kubernetes-aws-io --yes

AWS Commands

Use aws ec2 describe-availability-zones --region=us-east-1 to find out a region with 3+ AZs

rmccomb-bnet/kubernetes-federated-cluster