zapier/kubechecks

Schema validation always fails for CRDs

Closed this issue · 11 comments

bauerjs1 commented

Hey there,
unfortunately, I couldn't get the schema validation for CRDs to work.

  • If I stick to the default configuration, the /app/schemas directory in the container is empty and Kubechecks does not know about any CRDs. If there was any content from the container image, it is superseded by a volume mount, maybe due to this?
  • If I tell Kubechecks to use, e.g., 
    KUBECHECKS_SCHEMAS_LOCATION: https://github.com/datreeio/CRDs-catalog.git
    it successfully pulls from the repository, but still fails validation with 
    Error: external-secrets.io/v1beta1 ExternalSecret kubechecks - could not find schema for ExternalSecret
    (the schema for external-secrets.io/v1beta1/ExternalSecret is present in this repository)
  • I also tried using a custom git repository with all schema files in a flat structure but it still won't find anything
  • Mounting schemas from a volume to /app/schemas has no effect, either

Looking at the logs it seems to me that Kubechecks always searches for these schemas in the application source repos, no matter if I provide an absolute or relative path for KUBECHECKS_SCHEMAS_LOCATION:

WRN failed to find in-repo path error="stat /tmp/kubechecks-mr-clone3517385009/app/schemas: no such file or directory" location=/app/schemas

How can I make use of this feature? Is there an option to disable schema validation altogether?

EDIT: We're running kubechecks:v1.3.3

audrey-mux commented

Have you tried using a repo location? For example, I set KUBECHECKS_SCHEMAS_LOCATION to tools/kubechecks/schemas and it pulls everything it needs from there. The big caveat is that you gotta pull in the schemas you need.

bauerjs1 commented

I tried that, but it seems like the schemas would need to be present in every application repository instead of one central place, when providing a file path here. That is just not feasible in our case.

  • 👍1
audrey-mux commented

Ah, that makes sense. Being stuck with a giant monorepo makes it work.

  • 😄1
djeebus commented

Could it be a path issue with your schema repository? We use kubeconform under the hood, which expects us to pass a path template, not just a path. We take the value used in the env var you mentioned and add {{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json to it. This mimics the default schema repository, which ends up with files named similar to /v1.28.2/deployment-apps-v1.json. It's possible that this is the source of the issue, and I think we don't document it at all =/

bauerjs1 commented

@djeebus I tried that one, too. I used the URL from kubeconform docs but Kubechecks fails. From the logs, it looks like Kubechecks tries a git clone with that templated path instead of just passing it to kubeconform:

4:14PM ERR unable to clone repository, Cloning into '/tmp/schemas954146872'...
remote: 404: Not Found
fatal: repository 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json/' not found
 error="exit status 128"
4:14PM ERR failed to clone repository error="exit status 128" clone-url=https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json
4:14PM INF check done app=kubechecks-staging-gitlab check="validating app against schema" event_id=310 repo=gitlab result="Failed :red_circle:"

However, if I just specify a cloneable URL, Kubechecks pulls that successfully but then there is no path template for kubeconform to find the right schema (see second bullet point from OP).

bauerjs1 commented

Btw, I would also be fine if I could just disable kubeconform validations.

  • 👍1
bauerjs1 commented

@djeebus is there anything I can do to get rid of the failing pipelines?

djeebus commented

Sorry, been working on a big push to get multi-repo app-of-apps working. There are a few features that should help out here:

  • set the subdirectory of the schemas repo
  • disable schema validation
  • reduce schema validation errrors to warnings
  • customize schema path format

Not sure when we'll get those done, but does that sound like it'd help you out here?

bauerjs1 commented

Thanks @djeebus, sounds like any of these options might help mitigating the issue. The disable option might just be the quickest and easiest way in our case, since we don't need to rely on this feature atm, but they'd all be useful, I guess.

djeebus commented

If you upgrade to 1.5.1, we've made some improvements that might help:

  • KUBECHECKS_WORST_KUBECONFORM_STATE=warning: if kubeconform fails with something worse than a warning, reduce it to only a warning
  • fixed a bug that caused us to attempt to clone the url that you provided above, it should work now.
  • 🎉2
bauerjs1 commented

Great, now it works with

KUBECHECKS_SCHEMAS_LOCATION: https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json

Thanks again!

  • 🎉1