network.provider = kuberouter still tries to launch calico

[ineu]

Learning k0s/k0sctl, I'm trying to set up a new cluster, starting with a single machine.

1. On my local machine I've installed k0sctl 0.17.4
2. Created a config via k0sctl init --k0s > k0sctl.yaml
3. Updated the hosts section, hasn't changed anything else
4. Ran k0sctl apply, waited for it to report success, created kubeconfig, checked the pods

Result: kube-proxy and kube-router pods are up, coredns and metrics-server are stuck in the ContainerCreating state.
I've disabled systemd-resolved on the server, followed by a reboot, updated /etc/resolv.conf manually just in case, not luck.

Took a look at journalctl, it's full of the following:

Jan 14 10:45:39 hadrianus k0s[5083]: time="2024-01-14 10:45:39" level=info msg="E0114 10:45:39.637443 5308 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"46c74953-2110-4389-896b-7752ee7bc777\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"367f684987f7eaef3794f887346241be4bcf11ca558522eabf350d40d2289bbf\\\": plugin type=\\\"calico\\\" failed (delete): stat /etc/cni/net.d/calico-kubeconfig: no such file or directory\"" pod="kube-system/metrics-server-7556957bb7-lb2qh" podUID="46c74953-2110-4389-896b-7752ee7bc777"" component=kubelet stream=stderr

which I believe is the reason of the problem.

Rechecked the auto-generated k0sctl.yaml, it explicitly uses kuberouter, so no wonder there's no calico config.

        network:
          ...
          podCIDR: 10.244.0.0/16
          provider: kuberouter

Looks like calico is hardcoded somewhere which triggers the errors.

[ineu]

After k0sctl reset and manually removing /var/lib/calico dir (which I believe should also have been removed by the reset command?), I've run k0sctl apply again and now get another error, though similar:

Jan 14 10:58:46 hadrianus k0s[838]: time="2024-01-14 10:58:46" level=info msg="E0114 10:58:46.795104 1085 remote_runtime.go:222] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"ec028c23e07c9a30c9b07f1ef2b9b3bd050ed59279819dfdf6c1facfe1f4946e\": plugin type=\"calico\" failed (delete): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/" podSandboxID="ec028c23e07c9a30c9b07f1ef2b9b3bd050ed59279819dfdf6c1facfe1f4946e"" component=kubelet stream=stderr

[ineu]

Setting network.provider to calico seems to (partially) fix the issue: coredns pod is up, but metrics-server is not present in the list of pods at all.

[ineu]

Actually the same happens when I don't use k0sctl at all, running k0s directly on the server. I'll open a bug in https://github.com/k0sproject/k0s/issues instead

[ineu]

Couldn't reproduce the problem on a similar VM.