crossplane-contrib/provider-helm

Local insecure registry for oci based helm charts?

ramarnat opened this issue · 5 comments

ramarnat commented

What happened?

I have setup a local registry based on the instructions here - https://kind.sigs.k8s.io/docs/user/local-registry/

When setting the helm chart properties, using oci (the registry was loaded with the oci pkg):

        forProvider:
          namespace: cp
          chart:
            name: foo-operator
            repository: oci://registry:5000
            version: "1.7.0"
          insecureSkipTLSVerify: true

I have tried a bunch of different options, but none of them work. If we use a named registry the provider tries to use https:// , and produces this error:

 DEBUG   events  Warning {"object": {"kind":"Release","name":"foo","uid":"84508856-0b77-45a8-abee-2f898f7e0a7c","apiVersion":"helm.crossplane.io/v1beta1","resourceVersion":"8476"}, "reason": "CannotCreateExternalResource", "message": "failed to install release: failed to login to registry: Get \"https://registry:5000/v2/\": http: server gave HTTP response to HTTPS client"}

In kind the configuration for skipping tls is also set:

    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."registry:5000"]
          [plugins."io.containerd.grpc.v1.cri".registry.configs."registry:5000".tls]
            insecure_skip_verify = true
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
          endpoint = ["https://registry.k8s.io", "https://k8s.gcr.io"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5001"]
          endpoint = ["http://registry:5000", "oci://registry:5000"]

How can we reproduce it?

  1. Create a local registry following https://kind.sigs.k8s.io/docs/user/local-registry/
  2. Upload an oci helm chart
  3. Use the oci helm chart from a composition

What environment did it happen in?

Crossplane version:

Chart Name: crossplane
Chart Description: Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume.
Chart Version: 1.9.0
Chart Application Version: 1.9.0

Kube Version: v1.25.0
AbrohamLincoln commented

It appears as though you can perform a login to an insecure registry, but you cannot pull from an insecure registry.

If I do not provide credentials, the error message is as follows:

create failed: failed to install release: failed to login to registry: Get "http://registry:5000/v2/": no basic auth credentials

When credentials are provided, it appears to perform the login without issue, but errors on the pull:

 failed to install release: failed to pull chart: failed to do request: Head "https://registry:5000/v2/helm-charts/mychart/manifests/1.0.0": http: server gave HTTP response to HTTPS client

I do not know go, but it seems as though, the InsecureSkipTLSverify bool is not being passed to the Pull client.

Edit:
Looks like this may not even be possible until this is merged?
helm/helm#10408

AbrohamLincoln commented

Looks like this MR adds the missing functionality:
helm/helm#11599

jbw976 commented

Related to #202

jbw976 commented

I'm hoping this is either possible or closer to possible now that #202 has been resolved. If someone wants to try this scenario out and take it to the finish line if needed, that could be a welcome contribution 🙇

ramarnat commented

This probably does solve it, but I am not working with the original product that required having this issue resolved.