Support running Nova in-cluster?

[evandam]

Is your feature request related to a problem? Please describe.
We're looking into running Nova as a Kubernetes job on a regular basis to check for updates in our clusters. It seems like Nova is built under the assumption that it will always be run outside of a cluster where a user has a kube config/context setup.

I'm running on an EKS cluster (1.27) and using a serviceAccount with an admin ClusterRoleBinding, so permissions shouldn't be an issue as far as I can tell.

When running in a pod, I see the following logs:

F0810 16:15:29.825478       1 kube.go:62] error getting config with context : invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

However, setting KUBERNETES_MASTER=https://kubernetes.default.svc.cluster.local or something similar doesn't change anything (I'm assuming this is a generic log from the k8s SDK).

Describe the solution you'd like
Nova can attempt to connect in-cluster before using the kubeContext specified, or alternatively some --in-cluster flag we can pass.

Describe alternatives you've considered
Not too sure what options we have, but happy to hear your thoughts.

Additional context
Sadly I don't know Go very well and don't think I'd be able to open a PR 😞

Thank you for the work on this project, it's really really useful!

[sudermanjr]

Nova should definitely work in-cluster. We actually do this as part of our commercial software that utilizes nova reports. You can reference how we do this in our chart - https://github.com/FairwindsOps/charts/tree/master/stable/insights-agent. Here's an example of a templated cronjob from that chart.

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    linkerd.io/inject: disabled
    polaris.fairwinds.com/cpuLimitsMissing-exempt: "true"
    polaris.fairwinds.com/memoryLimitsMissing-exempt: "true"
    sidecar.istio.io/inject: "false"
  labels:
    app: insights-agent
  name: nova
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      activeDeadlineSeconds: 300
      backoffLimit: 1
      template:
        metadata:
          annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
          labels:
            app.kubernetes.io/name: nova
            app.kubernetes.io/part-of: insights-agent
        spec:
          containers:
            - command:
                - /nova
                - find
                - --helm
                - --containers
                - --config=/config/nova.yaml
                - -v3
              env: null
              image: quay.io/fairwinds/nova:v3.6
              imagePullPolicy: Always
              name: nova
              resources:
                requests:
                  cpu: 100m
                  memory: 128Mi
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false
                readOnlyRootFilesystem: true
                runAsGroup: 10324
                runAsNonRoot: true
                runAsUser: 10324
              volumeMounts:
                - mountPath: /output
                  name: output
                - mountPath: /config
                  name: config
                - mountPath: /tmp
                  name: tmp
            - command:
                - ./uploader.sh
                - --datatype
                - nova
                - --timeout
                - "300"
                - --organization
                - research-and-development
                - --cluster
                - sandbox
                - --host
                - https://insights.fairwinds.com
                - --version
                - v3.6
                - --file
                - /output/nova.json
              env:
                - name: FAIRWINDS_TOKEN
                  valueFrom:
                    secretKeyRef:
                      key: token
                      name: insights-agent-token
                - name: FAIRWINDS_AGENT_CHART_VERSION
                  value: 2.23.2
                - name: SEND_FAILURES
                  value: "true"
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              image: quay.io/fairwinds/insights-uploader:0.5.1
              imagePullPolicy: Always
              name: insights-uploader
              resources:
                limits:
                  cpu: 250m
                  memory: 512Mi
                requests:
                  cpu: 100m
                  memory: 128Mi
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false
                readOnlyRootFilesystem: true
                runAsGroup: 1000
                runAsNonRoot: true
                runAsUser: 1000
              volumeMounts:
                - mountPath: /output
                  name: output
          restartPolicy: Never
          serviceAccountName: insights-agent-nova
          volumes:
            - emptyDir: {}
              name: output
            - configMap:
                name: insights-agent-nova-config
              name: config
            - emptyDir: {}
              name: tmp
  schedule: 27 * * * *
  successfulJobsHistoryLimit: 1

[sudermanjr]

Also, is it possible you're not automounting the serviceAccountToken for that service account? Nova definitely needs that to access the kube api

[evandam]

@sudermanjr you're 100% right, user error as usual 😂

Appreciate the quick response!