/azure-consul-live-stream

"Secure Communication in a Meshy World" Live Stream

Primary LanguageHCLMozilla Public License 2.0MPL-2.0

Azure Consul Live Stream

This repository contains HashiCorp Terraform configuration and scripts used in the Azure Consul Live Stream.

Prerequisites

Clone the Github repository

git clone https://github.com/anubhavmishra/azure-consul-live-stream.git

Tutorial

This tutorial will be using the HashiCorp learn guide Secure Service Mesh Communication Across Kubernetes Clusters for setting up multi-cluster communication between two Kubernetes clusters running in AKS.

Generate Azure Credentials

Generate Azure client id and secret.

az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/YOUR_SUBSCRIPTION_ID"

Expected output:

{
  "appId": "00000000-0000-0000-0000-000000000000",
  "displayName": "azure-cli-2017-06-05-10-41-15",
  "name": "http://azure-cli-2017-06-05-10-41-15",
  "password": "0000-0000-0000-0000-000000000000",
  "tenant": "00000000-0000-0000-0000-000000000000"
}

appId - Client id. password - Client secret. tenant - Tenant id.

Export environment variables to configure the Azure Terraform provider.

export AZURE_CLIENT_ID="CLIENT_ID"
export AZURE_TENANT_ID="TENANT_ID"
export AZURE_CLIENT_SECRET="CLIENT_SECRET"
export AZURE_SUBSCRIPTION_ID="YOUR_SUBSCRIPTION_ID"
export TF_VAR_client_id=${AZURE_CLIENT_ID}
export TF_VAR_client_secret=${AZURE_CLIENT_SECRET}

Create AKS clusters

Create two AKS clusters in US East and US West.

cd aks
terraform apply

Export Kubernetes Config for the AKS clusters

Export the kubernetes config for the US East cluster

echo "$(terraform output aks_us_east_1_cluster_config)" > kubeconfig-us-east-1.yml

Export the kubernetes config for the US West cluster

echo "$(terraform output aks_us_west_1_cluster_config)" > kubeconfig-us-west-1.yml

Setup Kubernetes Config

Move the Kubernetes configuration files in the consul/ folder

cd consul/
mv ../aks/kubeconfig-us-* .

Merge Kubernetes configuration

touch kubeconfig-aks-clusters
export KUBECONFIG="kubeconfig-aks-clusters:kubeconfig-us-east-1.yml:kubeconfig-us-west-1.yml"

Validate both Kubernetes clusters

kubectl cluster-info --context="aks-east-1"

Expected output

Kubernetes master is running at https://akseast-2c0ce53f.hcp.eastus.azmk8s.io:443
CoreDNS is running at https://akseast-2c0ce53f.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://akseast-2c0ce53f.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
kubectl cluster-info --context="aks-west-1"

Expected output

Kubernetes master is running at https://akswest-fef70750.hcp.westus2.azmk8s.io:443
CoreDNS is running at https://akswest-fef70750.hcp.westus2.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://akswest-fef70750.hcp.westus2.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

Deploy Consul

Deploy Consul in US East.

helm install -f helm-consul-us-east-1-values.yaml consul hashicorp/consul --kube-context="aks-east-1" --wait

Watch for pods being created by using the following command.

watch -n 1 "kubectl get pods --context='aks-east-1'"

Check Consul members in the cluster.

kubectl --context='aks-east-1' exec statefulset/consul-server -- consul members

Note: Wait until the Consul service is ready. Use kubectl get services consul-ui --context="aks-east-1" to check if the EXTERNAL-IP is set.

View Consul UI

open https://$(kubectl get services consul-ui --context="aks-east-1" -o jsonpath="{.status.loadBalancer.ingress[0].ip}")

Deploy Services

Deploy frontend in US East.

kubectl apply -f kubernetes/frontend.yaml --context="aks-east-1"

Deploy api v1 in US East.

kubectl apply -f kubernetes/api-v1.yaml --context="aks-east-1"

Open another terminal window and call the frontend service.

kubectl get services frontend --context="aks-east-1"

See logs from frontend service.

kubectl --context="aks-east-1" logs -f frontend-x

See intentions in Consul UI

Create an intentions from frontend -> api to deny traffic.

Now we will do traffic splitting between api v1 and v2.

Traffic Splitting

Copy consul config to consul server for easy access

kubectl --context="aks-east-1" cp config/ consul-server-0:/

Create service defaults.

kubectl --context="aks-east-1" exec consul-server-0 -- consul config write /config/api-defaults.hcl

Deploy api v2 in US East.

kubectl apply -f kubernetes/api-v2.yaml --context="aks-east-1"

Create service resolver.

kubectl --context="aks-east-1" exec consul-server-0 -- consul config write /config/api-resolver.hcl

Create service splitter that splits traffic between v1 and v2.

kubectl --context="aks-east-1" exec consul-server-0 -- consul config write /config/api-splitter-50.hcl

Send the traffic to v2.

kubectl --context="aks-east-1" exec consul-server-0 -- consul config write /config/api-splitter-100.hcl

Multi-cluster

Uncomment federation from helm-consul-us-east-1-values.yaml

vim helm-consul-us-east-1-values.yaml

global:
  name: consul
  image: consul:1.8.0
  imageK8S: hashicorp/consul-k8s:0.16.0
  datacenter: us-east-1
  federation:
    enabled: true
    createFederationSecret: true
  tls:
    enabled: true
meshGateway:
 enabled: true
connectInject:
 enabled: true
ui:
 enabled: true
 service:
    enabled: true
    type: "LoadBalancer"

Run helm upgrade to update the US East cluster.

helm upgrade -f helm-consul-us-east-1-values.yaml consul hashicorp/consul --wait --kube-context="aks-east-1"

Note: This command takes about 3-5 mins to complete as we will need to restart all servers but it won't distrupt our service mesh communication.

The command will create a secret called consul-federation. To view the secret run the following command.

kubectl get secret consul-federation -o yaml

View the secret in the base64 encoded string

echo "eyJwcmltYXJ5X2RhdGFjZW50ZXIiOiJ1cy1lYXN0LTEiLCJwcmltYXJ5X2dhdGV3YXlzIjpbIjUyLjE0Ni40Mi42MDo0NDMiXX0" | base64 -d

Save the secret so we can apply to the US West cluster

kubectl get secret consul-federation -o yaml > consul-federation-secret.yaml

Open a new terminal window with US West cluster.

Apply the secret to US West cluster.

kubectl apply -f consul-federation-secret.yaml --context="aks-west-1"

Open the helm-consul-us-west-1-values.yaml file to view the US West cluster configuration.

vim helm-consul-us-west-1-values.yaml

global:
  datacenter: us-west-1
  image: consul:1.8.0
  imageK8S: hashicorp/consul-k8s:0.16.0
  tls:
    enabled: true
    caCert:
      secretName: consul-federation
      secretKey: caCert
    caKey:
      secretName: consul-federation
      secretKey: caKey
  federation:
    enabled: true
  name: consul
server:
  extraVolumes:
  - type: secret
    name: consul-federation
    items:
      - key: serverConfigJSON
        path: config.json
    load: true
connectInject:
  enabled: true
meshGateway:
  enabled: true

Install consul in US West cluster

helm install -f helm-consul-us-west-1-values.yaml hashicorp hashicorp/consul --wait --kube-context="aks-west-1"

Verify the installation

kubectl --context="aks-west-1" exec statefulset/consul-server -- consul members -wan

Apply the api-v2 application to US West

kubectl --context="aks-west-1" apply -f kubernetes/api-v2-us-west-1.yaml

Copy consul config to consul server since the servers restarted.

kubectl --context="aks-east-1" cp config/ consul-server-0:/

Delete the service resolver and splitter for api.

kubectl --context="aks-east-1" exec consul-server-0 -- consul config delete -kind service-resolver -name api
kubectl --context="aks-east-1" exec consul-server-0 -- consul config delete -kind service-splitter -name api

Show service failover by creating new service resolver and then deleting api v1 and v2 deployments.

kubectl --context="aks-east-1" exec consul-server-0 -- consul config write /config/api-failover-resolver.hcl

Next delete api v1 and v2

kubectl delete deployments api-v2 api-v1 --context="aks-east-1"

Show frontend logs that now point to US West.

kubectl --context="aks-east-1" logs -f frontend-x

Change frontend yaml config to include datacenter in the upstream service.

vim kubernetes/frontend.yaml

Apply the frontend deployment file to update the service in US East.

kubectl apply -f kubernetes/frontend.yaml --context="aks-east-1"

Show frontend logs that now point to US West.

kubectl --context="aks-east-1" logs -f frontend-x

Create api v1 and v2 services

kubectl apply -f kubernetes/api-v1.yaml --context="aks-east-1"
kubectl apply -f kubernetes/api-v2.yaml --context="aks-east-1"