`microk8s reset` does not completely reset the cluster

[LoniasGR]

Summary

When doing microk8s reset, validatingwebhookconfigurations.admissionregistration.k8s.io are not deleted. This can lead to weird errors, as mentioned also here. In my case, the problem was with the ingress-nginx-controller.

What Should Happen Instead?

microk8s reset should delete all previously resources, effectively providing a fresh cluster.

Reproduction Steps

1. Setup microk8s cluster.
2. Install ingress-nginx-controller as described here.
3. Run microk8s reset
4. Run microk8s.kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io

There should be still an ingress-nginx-admission webhook.

Introspection Report

I don't think it's needed.

Can you suggest a fix?

The problem is that here, from the non-namespaced API resources, only crds, priority classes, storage classes and namespaces are deleted. I'm not exactly sure which of the non-namepsaced API should be cleaned up, but probably there are more than just validating webhooks.

Are you interested in contributing with a fix?

Probably, if I was informed what resources should be actually deleted.

[ktsakalozos]

Hi @LoniasGR, the script that performs the reset is https://github.com/canonical/microk8s/blob/master/scripts/wrappers/reset.py . Indeed it may not be cleaning up everything. Often users prefer to snap remove and re-install as this might be faster if you run a lot of workloads.

[LoniasGR]

Thank you for the information. I created an MR that should probably solve the issue, since removing and re-installing is not a viable option for us.