Microk8s Cluster Tries to Pull Image from Insecure Private Registry With HTTPS

[Joseph-Hui]

Summary

I have a insecure private registry for used by a Microk8s cluster (3 master + 2 worker running on Ubuntu 22 VM that are communicating with a VPN network). When I try to create deployment with an image on the private registry, the request failed. I have done the following steps according to the How to work with a private registry and the following errors have be observed.

Failed to pull image "10.8.0.131/mynginx:registry": failed to pull and unpack image "10.8.0.131/mynginx:registry": failed to resolve reference "10.8.0.131/mynginx:registry": failed to do request: Head "https://10.8.0.131/v2/mynginx/manifests/registry": dial tcp 10.8.0.131:443: connect: connection refused

I tried with v1.29 and v1.30 and didn't succeed. About a week ago, I was able to run spark-submit to test the SparkPi program with the spark-py image on the same private registry. However, the Microk8s cluster was reset as I was learning and trying different setting.

What Should Happen Instead?

HTTP access should be used to pull the image from the insecure private registry and the deployment should succeed.

Failed to pull image "10.8.0.131/mynginx:registry": failed to pull and unpack image "10.8.0.131/mynginx:registry": failed to resolve reference "10.8.0.131/mynginx:registry": failed to do request: Head "https://10.8.0.131/v2/mynginx/manifests/registry": dial tcp 10.8.0.131:443: connect: connection refused

Reproduction Steps

1. Created hosts.toml on all master and worker nodes.

> sudo vi /var/snap/microk8s/current/args/certs.d/10.8.0.131/hosts.toml
server = “http://10.8.0.131"

[host.”http://10.8.0.131"]
capabilities = ["pull", “resolve"]

2. Create /etc/docker/daemon.json

{
	"insecure-registries":["10.8.0.131"]
}

3. Micok8s and docker.service were all restarted on all nodes.

4. Build the MyNgix image and deploy to Microk8s cluster

> docker build . -t 10.8.0.131/mynginx:registry 
> kubectl create deployment mynginx --image=10.8.0.131/mynginx:registry

Introspection Report

inspection-report.tar.gz

Can you suggest a fix?

1. A mechanism to check the setting of the private registry
2. A configuration to force to use insecure private registry

Are you interested in contributing with a fix?

I do not know how to fix.

[Joseph-Hui]

I added a self-signed SSL cert to the private registry and added the self-signed CA cert to the hosts.toml of the domain of the private registry. Images can be pulled from private registry to the cluster.