kubectl-incluster

I wrote this kubectl plugin in order to create a kubeconfig file out of an in-cluster configuration, i.e., using the mounted service account token and CA cert:

/var/run/secrets/kubernetes.io/serviceaccount/token
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Running kubectl incluster from inside a running pod will print a working kubeconfig that you can use somewhere else.

Content:

Use-case: Telepresence 1 + mitmproxy for debugging cert-manager
- Optional: read the Let's Encrypt jose+json payloads
Use-case: Telepresence 2 + mitmproxy for debugging cert-manager
Use-case: Telepresence 1 + mitmproxy for debugging the preflight agent
Use-case: mitmproxy inside the cluster (as opposed to using Telepresence 1)
Use-case: mitmproxy without kubectl-incluster
Use-case: mitmproxy to debug an admission webhook
kubectl-incluster manual
- The --print-client-cert flag
mitmproxy and Telepresence gotchas
- The $TELEPRESENCE_ROOT stays empty on Linux
Workaround for Google Kubernetes Engine (GKE)

Use-case: Telepresence 1 + mitmproxy for debugging cert-manager

In order to inspect the egress traffic coming from a Kubernetes controller (here, cert-manager), I want to be able to use mitmproxy through a Telepresence --run-shell session. For example, let's imagine you have a cert-manager deployment already running and that you want to see what requests it makes.

⚠️ Telepresence 1 (written in Python) has been "superseeded" by Telepresence 2 (written in Go). This use-case focuses on Telepresence 1. To see the same use-case using Telepresence 2, scroll down. Note that Telepresence 1 works better in certain scenarios (e.g., Telepresence 1 supports replacing Deployments that have runAsNonRoot: false set on them).

First, install Telepresence 1. To install it on Ubuntu:

curl -s https://packagecloud.io/install/repositories/datawireio/telepresence/script.deb.sh | sudo bash
sudo apt install --no-install-recommends telepresence

(see macOS instructions)

Then, install mitmproxy on both Linux and macOS:

brew install mitmproxy

The next step is to start mitmproxy. We will be using watch-stream.py, a script that makes sure the streaming GET requests are properly streamed by mitmproxy:

curl -L https://raw.githubusercontent.com/maelvls/kubectl-incluster/main/watch-stream.py >/tmp/watch-stream.py
mitmproxy -p 9090 --ssl-insecure -s /tmp/watch-stream.py --set client_certs=<(kubectl incluster --print-client-cert)

Note that we could avoid using --ssl-insecure by replacing it with something like:
mitmproxy -p 9090 --set ssl_verify_upstream_trusted_ca=<(kubectl incluster --print-ca-cert)
But since I don't run mitmproxy from inside the Telepresence shell, I don't have access to the $TELEPRESENCE_ROOT variable. So I don't bother and use --ssl-insecure instead.

Let us now install kubectl-incluser:

go install github.com/maelvls/kubectl-incluster@latest

Now, let's run cert-manager locally inside a Telepresence shell:

% git clone https://github.com/jetstack/cert-manager && cd cert-manager
% telepresence --namespace cert-manager --swap-deployment cert-manager --run-shell
T: Using a Pod instead of a Deployment for the Telepresence proxy. If you experience problems, please file an issue!
T: Forwarding remote port 9402 to local port 9402.
T: Connected. Flushing DNS cache.
T: Setup complete. Launching your command.

@boring_wozniak|bash-3.2$

Now, from this shell, let us run cert-manager:

HTTPS_PROXY=localhost:9090 go run ./cmd/controller --leader-elect=false --kubeconfig <(kubectl incluster --root $TELEPRESENCE_ROOT --replace-ca-cert ~/.mitmproxy/mitmproxy-ca.pem) -v=4

And TADA! We see all the requests made by our controller:

Optional: read the Let's Encrypt `jose+json` payloads

In mitmproxy, it is hard to read the JSON payloads sent by the ACME server since they are base64 encoded:

{
  "protected": "eyJhbGciOiJSUzI1Ni...E1MzY1Mjg1MCJ9",
  "payload": "eyJjc3IiOiJNSUlDblRDQ0FZ...EU3lHQ3BjLTlfanVBIn0",
  "signature": "qqYGqZDSSUwuLLxm6-...nygkb5S8igKPrw"
}

You can use the josejson.py script to decode the payload and protected fields "inline":

curl -L https://raw.githubusercontent.com/maelvls/kubectl-incluster/main/josejson.py >/tmp/josejson.py
mitmproxy -p 9090 -s /tmp/josejson.py

And now you can see the payload and protected fields "inline":

{
  "payload": {
    "csr": "MIICnTCCAYUCAQAwADCCAS...duroowkXh3tqgVFDSyGCpc-9_juA"
  },
  "protected": {
    "alg": "RS256",
    "kid": "https://acme-v02.api.letsencrypt.org/acme/acct/204416270",
    "nonce": "01017mM9r6R_TpKL-5zxAmMF5JmTCBI-v6AsLlGedj3pD1E",
    "url": "https://acme-v02.api.letsencrypt.org/acme/finalize/204416270/25153652850"
  },
  "signature": "qqYGqZDSSUwuLLxm6-...nygkb5S8igKPrw"
}

Use-case: Telepresence 2 + mitmproxy for debugging cert-manager

⚠️ As of 17 Sept 2021, Telepresence 2 does not support deployments configured with runAsNonRoot: false as per the issue #875. cert-manager, by default, uses runAsNonRoot: false, and you will see that Telepresence 2 hangs forever:

$ telepresence intercept cert-manager -n cert-manager -- bash
Launching Telepresence Root Daemon
Need root privileges to run: /home/mvalais/bin/telepresence daemon-foreground /home/mvalais/.cache/telepresence/logs /home/mvalais/.config/telepresence ''
[sudo] password for mvalais:
Launching Telepresence User Daemon
Connected to context k3d-boring (https://0.0.0.0:39767)
# ❌ Hangs forever.

To work around this, you need to change the securityContext of your cert-manager:

kubectl patch deploy cert-manager -n cert-manager --patch 'spec: {template: {spec: {securityContext: {runAsNonRoot: false}}}}'

This time, Telepresence 2 should work:

$ telepresence intercept cert-manager -n cert-manager -- bash
telepresence intercept cert-manager -n cert-manager -- bash
Using Deployment cert-manager
intercepted
    Intercept name    : cert-manager-cert-manager
    State             : ACTIVE
    Workload kind     : Deployment
    Destination       : 127.0.0.1:8080
    Volume Mount Point: /tmp/telfs-695026645
    Intercepting      : all TCP connections
mvalais@aorus:~/code/cert-manager$

Use-case: Telepresence 1 + mitmproxy for debugging the preflight agent

The preflight agent is a binary that runs in your Kubernetes cluster and reports information about certificates to the https://platform.jetstack.io dashboard. The free tier allows you to see if any of your certificates managed by cert-manager has an issue.

To debug the agent, the first step is to have the agent built:

git clone https://github.com/jetstack/preflight
cd preflight
make install

Then, you want to run telepresence:

telepresence --namespace jetstack-secure --swap-deployment agent --run-shell

Run the mitmproxy instance:

# In another shell, not in the telepresence shell.
mitmproxy -p 9090 --ssl-insecure --set client_certs=<(kubectl incluster --print-client-cert)

Finally you can run the agent:

🔰 Tip: to know which command-line arguments are used by a given deployment, you can use kubectl-args that extracts the args for the deployment. Imagining that you have ~/bin in your PATH, you can install it with:
cat <<'EOF' > /tmp/kubectl-args
#! /bin/bash
set -e -o pipefail
kubectl get deploy -ojsonpath='{.spec.template.spec.containers[0].args}' "$@" | jq -r '.[]' | awk '{if($2 != ""){print "\"" $0 "\""}else{print $0}}' |  tr '\n' ' '; echo
EOF
install /tmp/kubectl-args ~/bin
Then, use it with:
% kubectl args -n jetstack-secure agent
agent -c /etc/jetstack-secure/agent/config/config.yaml -k /etc/jetstack-secure/agent/credentials/credentials.json -p 0h1m0s

# Inside the telepresence shell.
HTTPS_PROXY=127.0.0.1:9090 KUBECONFIG=$(kubectl incluster --root $TELEPRESENCE_ROOT --replace-ca-cert ~/.mitmproxy/mitmproxy-ca.pem >/tmp/foo && echo /tmp/foo) preflight agent -c $TELEPRESENCE_ROOT/etc/jetstack-secure/agent/config/config.yaml -k $TELEPRESENCE_ROOT/etc/jetstack-secure/agent/credentials/credentials.json -p 0h1m0s

You will see:

Use-case: mitmproxy inside the cluster (as opposed to using Telepresence 1)

First, we need to have an instance of mitmproxy running:

kubectl apply -n jetstack-secure -f <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mitmproxy
  labels:
    app: mitmproxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mitmproxy
  template:
    metadata:
      labels:
        app: mitmproxy
    spec:
      containers:
        - name: mitmproxy
          image: mitmproxy/mitmproxy:latest
          args: [mitmweb, -p, "9090"]
          imagePullPolicy: Always
          ports:
            - containerPort: 8081
              name: ui
            - containerPort: 9090
              name: proxy
          resources:
            limits:
              memory: "460Mi"
              cpu: "200m"
---
kind: Service
apiVersion: v1
metadata:
  name: mitmproxy
spec:
  ports:
    - name: ui
      port: 8081
    - name: proxy
      port: 9090
  selector:
    app: mitmproxy
EOF

Then, let us see the mitmweb UI:

kubectl port-forward -n jetstack-secure $(kubectl get pod -n jetstack-secure -l app.kubernetes.io/name=agent -oname) 8081:8081

and head to http://localhost:8081.

Then, we need to add that to the running deployment that we want to debug:

kubectl edit deploy your-deployment

and add the following to the container's env:

spec:
  containers:
    - env:
        - name: HTTPS_PROXY
          value: http://mitmproxy:9090

⚠️ IMPORTANT ⚠️ : you also have to make sure the container's binary can disable TLS verification. Otherwise, no way to do that...

Use-case: mitmproxy without kubectl-incluster

Let us imagine we want to trace what kubectl get pods is doing under the hood.

First, let us work around the fact that Go binaries do not honor the HTTPS_PROXY variable for the 127.0.0.1 and localhost domains. Instead of 127.0.0.1, we will use the domain me:

grep "127.0.0.1[ ]*me$" /etc/hosts || sudo tee --append /etc/hosts <<<"127.0.0.1 me"

Then, let us make sure our system trusts Mitmproxy's root CA:

# Linux
sudo mkdir -p /usr/share/ca-certificates/mitmproxy
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/share/ca-certificates/mitmproxy/mitmproxy-ca-cert.crt
grep mitmproxy/mitmproxy-ca-cert.crt /etc/ca-certificates.conf \
  || sudo tee --append /etc/ca-certificates.conf <<<"mitmproxy/mitmproxy-ca-cert.crt"
sudo update-ca-certificates

# macOS
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/.mitmproxy/mitmproxy-ca-cert.pem

Let us start mitmproxy. We have to use --ssl-insecure due to the fact that we don't want to bother having mitmproxy to trust the apiserver. We need to give the correct client certificate to the proxy (if you are using a client certificate):

curl -L https://raw.githubusercontent.com/maelvls/kubectl-incluster/main/watch-stream.py >/tmp/watch-stream.py
kubectl config view --minify --flatten -o=go-template='{{(index ((index .users 0).user) "client-key-data")}}' | base64 -d >/tmp/client.pem
kubectl config view --minify --flatten -o=go-template='{{(index ((index .users 0).user) "client-certificate-data")}}' | base64 -d >>/tmp/client.pem
mitmproxy -p 9090 --ssl-insecure --set client_certs=/tmp/client.pem -s /tmp/watch-stream.py

Finally, let us run the command we want to HTTP-inspect:

HTTPS_PROXY=:9090 KUBECONFIG=<(kubectl config view --minify --flatten \
    | sed "s|certificate-authority-data:.*|certificate-authority-data: $(base64 -w0 < ~/.mitmproxy/mitmproxy-ca-cert.pem)|g" \
    | sed "s|127.0.0.1|me|") \
  kubectl get pods

🔰 The command kubectl config view --minify prints the kube config for the current context, which comes in very handy here.

Use-case: mitmproxy to debug an admission webhook

Unlike all the previous use-cases, we will be using mitmproxy in "reverse proxy" mode:

+------------------+                  +------------------+                  +--------------------+
|                  |                  |                  |                  |                    |
|                  |----------------->|                  |----------------->|                    |
|    apiserver     |                  |    mitmproxy     |                  |cert-manager-webhook|
|                  |                  |      :8080       |                  |       :8081        |
|                  |                  |                  |                  |                    |
+------------------+                  +------------------+                  +--------------------+
                                         reverse proxy

We want to be able to see what the apiserver is sending to cert-manager webhook.

To do that, we will be running the webhook out-of-cluster to make it easier when running mitmproxy. We could be using kubetap that does a similar job, but it does not support setting your own CA certificate to be served, meaning that we can't have a way to make the apiserver trust the webhook.

$ telepresence intercept cert-manager-webhook -n cert-manager -- bash
Using Deployment cert-manager-webhook
intercepted
    Intercept name    : cert-manager-webhook-cert-manager
    State             : ACTIVE
    Workload kind     : Deployment
    Destination       : 127.0.0.1:8080
    Volume Mount Point: /tmp/telfs-584691868
    Intercepting      : all TCP connections

Anything that hits cert-manager-webhook.cert-manager.svc:443 will be forwarded to the host on 127.0.0.1:8080.

We now need to force the apiserver to trust mitmproxy:

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: cert-manager-webhook-ca
  namespace: cert-manager
data:
  ca.crt: "$(cat ~/.mitmproxy/mitmproxy-ca-cert.pem | base64 -w0)"
  tls.crt: "$(cat ~/.mitmproxy/mitmproxy-ca-cert.pem | base64 -w0)"
  tls.key: "$(cat ~/.mitmproxy/mitmproxy-ca.pem | base64 -w0)"
EOF

cert-manager-cainjector will then take care of stuffing the above ca.crt into the caBundle in all cert-manager CRDs so that the apiserver knows how to verify the TLS certificate handled by mitmproxy.

Now, let's run mitmproxy on port 8080:

mitmproxy -p 8080 --mode reverse:https://localhost:8081 --ssl-insecure

Finally, we can run the webhook on port 8081:

go run ./cmd/webhook/ webhook --v=2 --secure-port=8081 --dynamic-serving-ca-secret-namespace=cert-manager \
  --dynamic-serving-ca-secret-name=cert-manager-webhook-ca \
  --dynamic-serving-dns-names=cert-manager-webhook,cert-manager-webhook.cert-manager,cert-manager-webhook.cert-manager.svc \
  --kubeconfig=$(kubectl incluster >/tmp/in.pem && echo /tmp/in.pem)