/k8s.vault_helm_deploy

Hashicorp Vault Kubernetes setup using Helm

Vault Helm Setup on Kubernetes

This is a draft guide to vault-helm setup & running as well as its configuration with some kubectl examples.

demo

Monitoring Kubernetes

Monitoring the current kubernetes status log or details is possible by way of CLI kubectl (with tools like k9s or just watch being helpful) or the UI kubernetes dashboard.

A separate terminal / shell session would be good to have at hand for the purposes of monitoring which can commonly be achieved with some flag (--watch / '-w', -f, etc) to kubectl where its supported or otherwise via the invoking of watch ... interval updates.

kubectl cluster-info ;
# we have cluster access & confirmed `kubectl cluster-info` is working.
# with gke for example you may need to redo: `gcloud init && cloud container clusters get-credentials CLUSTER_NAME ;`
# with aws: `aws ...`, etc...

kubectl logs vault-0 -f ;  # // follow logs of pod vault-0
kubectl get deployments --all-namespaces --watch ;  # // logs of deployments
kubectl get events --all-namespaces --sort-by=".metadata.creationTimestamp" --watch ; #  // all events

An alternate approach that may be easier and useful for familiarisation or quick navigation is the CLI tool k9s which helps in switching to specific components of kubernetes:

brew install derailed/k9s/k9s ;
# // for linux: https://github.com/derailed/k9s/releases

k9s ;
# // in k9s press '0' to show all namespaces
# // in k9s CTRL+a allows for quick switch from default pod via
# CTRL+c when done ;
# // you can keep this open to view change in pods or view logs like

kubernetes dashboard setup

The Kubernetes dashboard UI can be installed & configured as demonstrated below. A Kubernetes dashboard may already be provided to you by way of your providers web console / panel (for example in GKE) and you may not need this step.

# // for browser UI on 'http://localhost:8001/ui...':
# // install kubernetes dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml ;

# // ---------------------------------------------------------------------------
# Setup dashboard for cloud managed-kubernetes: EKS, AKS, GKE, etc.
# On GKE we can get UI credentials like:
gcloud config config-helper --format=json | jq -r '.credential.access_token' ;
# // ^^ GKE users will need this token for k8s-dashboard ui login

# // ---------------------------------------------------------------------------
# // DIY Clusters need setup (ditto some cloud providers)
# // get rbac from: https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md#admin-privileges
#kubectl apply -f rbac.yaml ;  # // may also need SA
#kubectl -n kubernetes-dashboard get secret | grep token ; # get INSTANCE_NAME
#kubectl -n kubernetes-dashboard describe secrets kubernetes-dashboard-token-INSTANCE_NAME ;
#kubectl config set-credentials cluster-admin --token='......' ;

# // ---------------------------------------------------------------------------
kubectl proxy ;
# // In Browser go to using token copied or use kubeconfig file:
# http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login

url: kubernetes dashboard

Installation & Configuration

Helm Setup

# // on macOS:
brew install helm ;
# curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get-helm-3 > get_helm.sh
# chmod 700 get_helm.sh ;
# ./get_helm.sh ;  # Linux

Vault install

git clone https://github.com/hashicorp/vault-helm.git ;
cd vault-helm ;
helm install vault ./ ;

kubectl status of vault

kubectl get pods --all-namespaces | grep vault ;
 # NAMESPACE     NAME                                                       READY   STATUS    RESTARTS   AGE
 # default       vault-0                                                    0/1     Running   0          116s
 # default       vault-agent-injector-5b9c8655f9-lm4rh                      1/1     Running   0          117s

kubectl logs vault-0 ;
 # 2020-04-02T11:01:55.353Z [INFO]  core: seal configuration missing, not initialized
 # 2020-04-02T11:01:58.343Z [INFO]  core: seal configuration missing, not initialized
 # ...
 # // this is because we've not updated default helm vaules.yaml

kubectl get deployment --all-namespaces | grep vault ;
 # NAMESPACE     NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
 # default       vault-agent-injector                       1/1     1            1           2m

kubectl get svc --all-namespaces | grep vault ;
 # NAMESPACE     NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
 # default       vault                      ClusterIP   10.97.10.231   <none>        8200/TCP,8201/TCP   2m
 # default       vault-agent-injector-svc   ClusterIP   10.97.9.210    <none>        443/TCP             2m
 # default       vault-internal             ClusterIP   None           <none>        8200/TCP,8201/TCP   2m

kubectl get sa --all-namespaces ;
# // service accounts

kubectl get events  --all-namespaces --sort-by=".metadata.creationTimestamp" ;
# // as above with `--watch ;` # can be helpful 

Vault Configure for Development Mode

Make all needed adjustments to the values.yaml of the vault-helm package.

# // we want to edit values.yaml to allow for eg: dev
# nano values.yaml ;

helm delete vault ;  # // delete previous helm deployment

# // if you want to remove pv to start anew:
kubectl get pvc --all-namespaces ;

kubectl delete pvc data-vault-0 data-vault-1 data-vault-2 ;  # // ...
# // otherwise you'll have to 'rm -rf vault/data/' from within pods to reset.

helm install vault ./ ;  # // re-install
kubectl logs vault-0 ;  # // should have root tokens now in dev mode & unsealed.
 # ==> Vault server configuration:
 # 
 #              Api Address: http://10.32.1.6:8200
 #                      Cgo: disabled
 #          Cluster Address: https://10.32.1.6:8201
 #               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
 #                Log Level: info
 #                    Mlock: supported: true, enabled: false
 #            Recovery Mode: false
 #                  Storage: inmem
 #                  Version: Vault v1.3.3
 # 
 # 2020-04-02T12:50:15.778Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
 # 
 # # // ...
 # 
 # The unseal key and root token are displayed below in case you want to
 # seal/unseal the Vault or re-authenticate.
 # 
 # Unseal Key: ......
 # Root Token: root
 # 
 # # // ...
 #

# // for CLI & UI browser access lets port-forward:
kubectl port-forward service/vault 8200:8200 ;  # // redo if redeployed.
# http://127.0.0.1:8200/ui/vault/auth?with=token

Vault Configure for HA Mode with Integrated Raft Storage & UI Enabled

Adjust values.yaml to include HA + raft storage.

# // we want to edit values.yaml
# nano values.yaml ;

helm delete vault ;  # // delete previous helm deployment

# // if you want to remove pv to start anew:
kubectl get pvc --all-namespaces ;

kubectl delete pvc data-vault-0 data-vault-1 data-vault-2 ;  # // ...
# // otherwise you'll have to 'rm -rf vault/data/' from within pods to reset.

helm install vault ./ ;  # // re-install

# // -----------------------------------------------------
# // NODE0 - inline CLI on vault-0 pod:
kubectl exec -it vault-0 -- /bin/vault 'status' ;
# // ... status from pod vault-0

# // NODE0 - shell attach to vault-0 pod:
kubectl exec -it vault-0 env PS1="\u@\h:\w$ " -- /bin/sh ;
# ...
# vault@vault-0:/$ \
vault operator init -key-shares=1 -key-threshold=1 ;  # // VUSEAL1=... populate.
 # Unseal Key 1: B599eB74XhWWTjQN9f2ttIkjZd7RfUQT6Ag+voPgpvg=
 # 
 # Initial Root Token: s.pznxZBwhKyiByMjab2zLFyV3
# ...
# vault@vault-0:/$ \
VUSEAL1='......' && vault operator unseal $VUSEAL1 ;
 # ...
 # Initialized            true
 # Sealed                 false
# ...

# vault@vault-0:/$ \
exit ;  # // or CTRL+D on keyboard


# // -----------------------------------------------------
# // local machine / operator
export VAULT_ADDR='http://0.0.0.0:8200' ;
vault status ;
# // should show the same as earlier kubectl exec version of status.


# // -----------------------------------------------------
# // NODE1 - shell attach to vault-1 pod:
kubectl exec -it vault-1 env PS1="\u@\h:\w$ " -- /bin/sh ;
# vault@vault-1:/$ \
vault operator raft join http://vault-0.vault-internal:8200 ;
 # Key       Value
 # ---       -----
 # Joined    true

# vault@vault-1:/$ \
VUSEAL1='......' && vault operator unseal $VUSEAL1 ;
 # ...
 # Initialized            true
 # Sealed                 false

# vault@vault-1:/$ \
exit ;  # // or CTRL+D on keyboard


# // -----------------------------------------------------
# // NODE2 - shell attach to vault-2 pod:
kubectl exec -it vault-2 env PS1="\u@\h:\w$ " -- /bin/sh ;
# vault@vault-2:/$ \
vault operator raft join http://vault-0.vault-internal:8200 ;
 # Key       Value
 # ---       -----
 # Joined    true

# vault@vault-2:/$ \
VUSEAL1='......' && vault operator unseal $VUSEAL1 ;
 # ...
 # Initialized            true
 # Sealed                 false

# vault@vault-2:/$ \
exit ;  # // or CTRL+D on keyboard


# // -----------------------------------------------------
# // local machine / operator
VTROOT='s.pznxZBwhKyiByMjab2zLFyV3' ;  # // SET your root token from earlier
VAULT_TOKEN=$VTROOT vault operator raft configuration -format=json
# // ...
# // JSON of raft peers currently configured

Reference