Monitor etcd for k0s

[onedr0p]

Looks like it requires grabbing the content of these files and applying a secret for kube-prometheus-stack can use:

curl --cacert /var/lib/k0s/pki/etcd/ca.crt --cert /var/lib/k0s/pki/etcd/server.crt --key /var/lib/k0s/pki/etcd/server.key https://localhost:2379/metrics

[onedr0p]

@brunnels I wonder if we could use a hook in the k0s-config to put these into a secret in the cluster, thoughts?

hooks:
  apply:
    after:
      - sudo k0s kubectl create namespace observability > /dev/null 2>&1 || true
      - sudo k0s kubectl --namespace observability create secret generic etcd-certs --from-file=cacert=/var/lib/k0s/pki/etcd/ca.crt --from-file=cert=/var/lib/k0s/pki/etcd/server.crt --from-file=key=/var/lib/k0s/pki/etcd/server.key  > /dev/null 2>&1 || true

Actually it looks like these need to be pulled in from disk in the kps etcd servicemonitor, tlsConfig does support pulling from secrets or configmaps but the Chart doesn't support it so it would need a postRenderer patch on the kps HelmRelease if we wanted to do that.

postRenderers:
  - kustomize:
      patches:
        - patch: |
            - op: replace
              path: /spec/endpoints/0
              value:
                bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
                port: http-metrics
                tlsConfig:
                  keySecret:
                    key: k0s-etcd-certs
                    name: key
                  cert:
                    secret:
                      key: k0s-etcd-certs
                      name: cert
                  ca:
                    secret:
                      key: k0s-etcd-certs
                      name: ca
          target:
            kind: ServiceMonitor
            name: kube-prometheus-stack-kube-etcd

Nevermind, all this is mute. etcd is only listening on 127.0.0.1:2379 which means prometheus cannot scrape it.

[onedr0p]

Solution was much easier than I thought
#1179