pods running in kube-system are down when using lxe
psamadda opened this issue · 7 comments
Using kubespray I have formed k8s cluster. The cluster works fine with containerd as container runtime endpoint. But if I change the endpoint to use lxe, all the pods of kube-system are going down and claiming that the images are missing. If I try to create a new pod it says the image alias is missing. I want all the containers of kube-system namespace use containerd image and my created pod uses lxe image. Is it possible?
lxd-remote-config is using the following config.yml
default-remote: local
remotes:
images:
ubuntu:
addr: Ubuntu Cloud Images
public: true
protocol: simplestreams
…
My pod creation config content:
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
containers:
name: nginx
image: “ubuntu/xenial/amd64”
…
lxe executes purely on what lxd provides, and lxd does not offer OCI images support. If you want to use purely lxc as your runtime, you can use lxcri which is maintained by the lxc developers. lxc can use OCI using oci-template, but lxd doesn't use this feature. The defined images in the kube-system pods really don't exist.
Your pod.yaml actually looks fine. I don't know if it was a copy-paste issue from my side or due to github-rendering, but when I used your pod description as base, the double-quotes in image were wrong: they were “ instead of ". Have a quick look at that - if it's not the double-quotes issue, can you paste me your pods status after creating it?
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
image: "ubuntu/xenial/amd64"
tolerations:
- key: my.example.com/containerruntime
operator: Exists(obviously I adapted the toleration to fit our cluster)
Above pod produces:
$ kubectl get pod nginx -o yaml
apiVersion: v1
kind: Pod
[...]
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-05-16T16:51:46Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-05-16T16:52:08Z"
message: 'containers with unready status: [nginx]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-05-16T16:52:08Z"
message: 'containers with unready status: [nginx]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-05-16T16:51:47Z"
status: "True"
type: PodScheduled
containerStatuses:
- image: ubuntu/xenial/amd64
imageID: ""
lastState: {}
name: nginx
ready: false
restartCount: 0
started: false
state:
waiting:
reason: ContainerCreating
hostIP: 192.168.12.40
phase: Pending
podIP: 192.168.12.95
podIPs:
- ip: 192.168.12.95
qosClass: BestEffort
startTime: "2022-05-16T16:51:46Z"
(It is currently pulling the image)
If you want to use existing OCI containers, like the ones sitting in kube-system, and also want use lxe, you could split the container runtime per node. E.g. A node is running all OCI images using containerd (or cri-o or lxcri or whatever) and another node is running all lxd images using lxe. This can be done by tainting the nodes. I've assigned a taint containerruntime=lxd to a node, now only pods which "tolerate" this taint (like you can see above in my pod.yaml) can run on this node, which you would do for all lxd based pods.
This will make the pods run, but networking will be your next step to solve (since you can't use the OCI images from the usual CNI providers), but we can get to that later.
double-quote issue is copy-paste issue. I have created the pod and the output is attached. It says the image alias is not found. I have also attached the state of the pods in kube-system namespace.
output.txt
Looks like those pods are coming from a daemonset. Again, LXD (and thus LXE) can't run docker containers (OCI images). You have to taint that node to prevent those pods from beeing assigned to that node. The image field in those pods usually refer to an image in hub.docker.com, which are OCI images and thus will never work.
Regarding alias issue: Can you update your initial post to formatted code? I can't verify your config as in yaml the indentions are part of the syntax.
Okay, I've found the cause of the second issue (alias not found). Ignore what I asked about the yaml format. Although yours still looks off from what I can see, as there is a "images:" property that shouldn't be there. Also, the internal LXD client will automatically add some of the default remotes implicitly, so they don't have to actually exist in that file anyway.
Thank you for your input. You gave me the reason to write a full step by step guide to setup a multi node kubernetes cluster which uses LXE on one node. In my case I use k3s, as it is slim and can be deployed anywhere (even locally) and does not require a cloud provider. Have a look, it might have some useful details: https://github.com/automaticserver/lxe/blob/update_deps/doc/examples/multi-node-cluster-with-k3s.md (Future readers: this guide will be merged to master once bug fix is complete)
The problem is lxd 5.0/5.1, they changed the error message when trying to resolve an alias from "not found" to "Image alias not found" a little while ago. LXE has to check if the provided image string is a valid alias or fingerprint and the not found type is now different, resulting in wrong behaviour. It was already bugging me for a while that LXD didn't define error types for the different cases. It is a bit better now but not as detailed as I would've hoped, but I can now check against it.
You will hear from me once resolved. This will probably also get a new tag release.
Thanks for looking into it. I followed https://github.com/automaticserver/lxe/blob/update_deps/doc/examples/multi-node-cluster-with-k3s.md with a little bit different path. I have two different VMs(Not like your case of VMs created in same box). Everything worked successfully but the pod created in node2 complains the issue of image not found. Please have a look into the attachments.
Sorry for letting you wait - I have fixed this issue. But I also wanted to clean up some responsibilities in the packages and this task was bigger than expected (and some holidays in between). I have improved maintenance of the code a lot this way.
I have prepared a prerelease with all the improvements, you can install it using: go install github.com/automaticserver/lxe/cmd/lxe@v0.4.0-rc3 or the prebuilt binaries from GHA via the download section of the tag: https://github.com/automaticserver/lxe/releases/tag/v0.4.0-rc3
I also finished the example guide with some hints about the next steps regarding container networking.
All further developments until final release will be handled in #17.
Thanks for looking into it. It works fine.