anchore/k8s-inventory

Embedded kai not returning inventories running on helm

Closed this issue · 6 comments

dspalmer99 commented

What happened:
Running enterprise with KAI embedded (as in the test case here), I am able to add a cluster (in this case, a reference to the enterprise cluster itself), but when enterprise periodically calls kai, it gets no results. I was able to reproduce this manually by connection to the catalog container and running kai from /usr/bin, which returns no results.

What you expected to happen:
Inventory results to be returned (as seen in the container logs and by querying GET /v1/enterprise/inventories

How to reproduce it (as minimally and precisely as possible):

  • Start enterprise via helm using the values file here
  • Setup port forwarding for the api container:
kubectl port-forward svc/test-release-anchore-engine-api 8228:8228
  • Follow the instructions in the test case to create a cluster config, and add it to the call to POST /v1/enterprise/inventories/clusters in the postman collection referenced in the test case.
  • Look at the logs for the catalog container and confirm the system is periodically querying for inventory details (current default is 5 minutes)
  • Call GET /v1/enterprise/inventories via cli or (more easily) the aforementioned postman collection. An empty array will be returned.

Anything else we need to know?:
I tried this running enterprise via helm locally both on top of docker desktop and minikube. Neither worked. In talking with @dakaneye it may be a case that kai running embedded in enterprise cannot resolve the host. The cluster config for the API is specific to my machine, but the cluster_server values I used were:

  • docker desktop cluster:
"cluster_server": "https://kubernetes.docker.internal:6443"
  • minikube:
"cluster_server": "https://127.0.0.1:55020"

Environment:

  • kai version (use kai version):
  • OS (e.g: cat /etc/os-release or similar):
dspalmer99 commented

@rbrady and I were looking at much of this together. Ryan, any details I missed?

dakaneye commented

I think this can be closed, pending the instructions I sent over are determined to be working, basically, when Anchore is running within Kubernetes, the cluster_server needs to set based on the value of the environment variable KUBERNETES_SERVICE_HOST (from within the catalog container), and the credential set based on the service-account token in /var/run/secrets/kubernetes.io/serviceaccount/token

Follow up should certainly be to improve the error messaging when authentication or connection fails to the k8s api

dspalmer99 commented

Yep, @rbrady and I both confirmed this is working as expected now with the config instruction changes you sent us, so I will close this.

bhearn7 commented

@dspalmer99 @rbrady @dakaneye I could not get embedded kai working either on 3.1.0 (no data even though images had been scanned)... could it be because I set .Values.anchoreCatalog.createServiceAccount: true before sending the kai POST request?

$ anchorectl -u admin -p foobar --url https://anchore-api.bigbang.dev inventory list
IMAGE TAG                                                                                     IMAGE DIGEST  CONTEXT                         LAST SEEN AT          INVENTORY TYPE 
docker.io/bitnami/postgresql:11.13.0-debian-10-r12                                            unknown       k3d-k3s-default/postgres        2021-09-08T08:26:50Z  kubernetes      
docker.io/rancher/coredns-coredns:1.8.3                                                       unknown       k3d-k3s-default/kube-system     2021-09-08T08:26:50Z  kubernetes      
docker.io/rancher/klipper-lb:v0.2.0                                                           unknown       k3d-k3s-default/istio-system    2021-09-08T08:26:50Z  kubernetes      
docker.io/rancher/local-path-provisioner:v0.0.19                                              unknown       k3d-k3s-default/kube-system     2021-09-08T08:26:50Z  kubernetes      
docker.io/rancher/metrics-server:v0.3.6                                                       unknown       k3d-k3s-default/kube-system     2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/anchore/enterprise/enterprise:3.1.0                                unknown       k3d-k3s-default/anchore         2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/anchore/enterpriseui/enterpriseui:3.1.0                            unknown       k3d-k3s-default/anchore         2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/bitnami/redis:6.2.2                                                unknown       k3d-k3s-default/anchore         2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/fluxcd/helm-controller:v0.11.0                                     unknown       k3d-k3s-default/flux-system     2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/fluxcd/kustomize-controller:v0.13.0                                unknown       k3d-k3s-default/flux-system     2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/fluxcd/notification-controller:v0.15.0                             unknown       k3d-k3s-default/flux-system     2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/fluxcd/source-controller:v0.14.0                                   unknown       k3d-k3s-default/flux-system     2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/kiwigrid/k8s-sidecar:1.10.6                                        unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/coreos/kube-state-metrics:v1.9.8                        unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/grafana/grafana:7.5.2                                   unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/istio/operator:1.10.4                                   unknown       k3d-k3s-default/istio-operator  2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/istio/pilot:1.10.4                                      unknown       k3d-k3s-default/istio-system    2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/istio/proxyv2:1.10.4                                    unknown       k3d-k3s-default/twistlock       2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/istio/proxyv2:1.10.4                                    unknown       k3d-k3s-default/istio-system    2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/postgres/postgresql96:9.6.18                            unknown       k3d-k3s-default/anchore         2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-config-reloader:v0.46.0  unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-operator:v0.46.0         unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/prometheus/alertmanager:v0.21.0                         unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/prometheus/node-exporter:v1.0.1                         unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/opensource/prometheus/prometheus:v2.25.0                           unknown       k3d-k3s-default/monitoring      2021-09-08T08:26:50Z  kubernetes      
registry1.dso.mil/ironbank/twistlock/console/console:21.04.439                                unknown       k3d-k3s-default/twistlock       2021-09-08T08:26:50Z  kubernetes 
$ anchorectl -u admin -p foobar --url anchore-api.bigbang.dev compliance list
No Data found
bhearn7 commented

Seeing the same behavior unfortunately when deploying KAI via helm. Logs look good, but no data from anchorectl or the UI

bhearn7 commented

please disregard my comments - my issue seems to stem from the containerd runtime and this PR: #27