Capi Ubuntu 22.04 image built with Qemu fails to bootstrap correctly

[nikParasyr]

What steps did you take and what happened:

• Built an ubuntu 22.04 image using the qemu provider
• (came across the issue mentioned in #1076 but overcame it )
• upload the image to an Openstack cluster
• try and deploy a new cluster using that image
• the first controlPlane boots but never gets ready, ssh is not working and k8s is not initialized
• i was able to get access to the vm through Openstack console and booting to single user mode from linux grub, while trying to figure out what was wrong I noticed:
  • that the kubeadm config and kubernetes pki files are not copied to the vm
  • I then looked through the various cloud-init services via journalctl but could not find a clear "error" showing some of them failing
  • I double-checked the secret created on the management cluster containing the cloud-init for the node and the openstack user_data of the vm (visible from inside the vm using the metadata service). Both of the seemed were available and seemed correct.
  • Not a cloud-init guru here, but i think cloud-final is the one responsible for copying these files etc, but i couldnt find in its logs were/why it failed

What did you expect to happen:
The node to get bootstrapped correctly

Anything else you would like to add:

• ubuntu 20.04 images built with the qemu provider worked just fine in the past -- first image with ubuntu 22.04
• using the openstack remote client builts the image correctly and i can deploy the clusters
• ubuntu 22.04 images created with qemu are quite big.... 10-15G each
• also tried 3 different base images and all of them failed

Environment:

Project (Image Builder for Cluster API, kube-deploy/imagebuilder, konfigadm):

Additional info for Image Builder for Cluster API related issues:

• OS (e.g. from /etc/os-release, or cmd /c ver):
• Packer Version:
• Packer Provider:
• Ansible Version:
• Cluster-api version (if using): 1.4.1
• Kubernetes version: (use kubectl version): 1.26.4

/kind bug

[chess-knight]

Hello @nikParasyr,
introduced #1146 which I tested on OpenStack and it worked for me. Can you try your setup with these changes to see if it works too?
Note: I experienced the same 10-15G big images

[ffais]

Hi @chess-knight,
I tried with the changes present in your pull request, but the problem with the qemu is still present.

[chess-knight]

I tried with the changes present in your pull request, but the problem with the qemu is still present.

Hi @ffais
I retest it and for me, it works.
I used these steps:

1. build qemu ubuntu 22.04 image(kubernetes v1.26.3):

PACKER_FLAGS="--var 'kubernetes_rpm_version=1.26.3-0' --var 'kubernetes_semver=v1.26.3' --var 'kubernetes_series=v1.26' --var 'kubernetes_deb_version=1.26.3-00'" mak
e build-qemu-ubuntu-2204

2. create the image in the OpenStack(it is still big 12.12 GB):

openstack image create --disk-format qcow2 --min-ram 1024 --min-disk 20 --file output/ubuntu-2204-kube-v1.26.3/ubuntu-2204-kube-v1.26.3 test-capi-image-v1.26.3

3. use it with the CAPO(capi v1.3.5, capo v0.7.1):

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: chess-knight-chesscluster-control-plane-genc01
  namespace: default
spec:
  template:
    spec:
      flavor: SCS-2V:4:20
      image: test-capi-image-v1.26.3
      ...

Note: Flavor SCS-2V:4:20 is 2 VCPUS, 4GB RAM and 20GB Disk

[ffais]

Hi @chess-knight,
Now it's working for me too, I've also tested with the latest Ubuntu image 22.04.2. Using the lastest Ubuntu image you can save time during build process.

[chess-knight]

Now it's working for me too, I've also tested with the latest Ubuntu image 22.04.2.

Hi @ffais,
thank you for the successful testing.

I also created an issue #1155 for big qcow2 images, so we will not forget it.