using clone

[travisghansen]

I'm interested in making this more cloud-like utilizing clone functionality of base images + cloud-init to inject keys etc.

Here's a crude outline of manual steps I did that I think will do the trick:

# clone
# -description
# -pool
pvesh create /nodes/cloud01/qemu/9003/clone --output-format json-pretty -newid 114

create linked clone of drive scsi0 (bitness-nfs:9003/base-9003-disk-0.qcow2)
clone 9003/base-9003-disk-0.qcow2: images, vm-114-disk-0.qcow2, 114 to vm-114-disk-0.qcow2 (base=../9003/base-9003-disk-0.qcow2)
Formatting '/mnt/pve/bitness-nfs/images/114/vm-114-disk-0.qcow2', fmt=qcow2 size=2361393152 backing_file=../9003/base-9003-disk-0.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
create full clone of drive ide2 (bitness-nfs:9003/vm-9003-cloudinit.qcow2)
Formatting '/mnt/pve/bitness-nfs/images/114/vm-114-cloudinit.qcow2', fmt=qcow2 size=4194304 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
"UPID:cloud01:00001DED:09D28645:5F1325C9:qmclone:9003:root@pam:"

# resize disk
pvesh set /nodes/cloud01/qemu/114/resize --output-format json-pretty -disk scsi0 -size 20G
Image resized.

# sockets, memory, etc
# -sshkeys (1 per line)
# -cores (# of cores per socket)
# -cpu (set cpu details)
pvesh create /nodes/cloud01/qemu/114/config --output-format json-pretty -autostart 1 -memory 4096 -sockets 4


# get currently set sshkeys 
pvesh get /nodes/cloud01/qemu/114/config --output-format json-pretty

# append newly created key to list
# urlencoded, 1 per-line
pvesh create /nodes/cloud01/qemu/114/config --output-format json-pretty -sshkeys ssh-rsa%20AAAAB3NzaC1yc2EAAAABIwAAAQEAnXCCFFYODq4Q%2FXy2NxUqbFWPm3nvK6NZezdWQCDI5gJrHJ%2Bw5%2BqemAQiLGZEVY0mfPrZk4bv%2FFUvJdF4ajq%2BQnFJ%2FnKAUQP0YI5QBQzAWWk4N8jLxBj9OjZ91tx9P3iV4qZ2V2gteQRQtnme6kojfgwajBhe1cFugkYEWVZIu08laixoUpq015j4JDt2bu0uiDbxG7XEze3kk5R23ynFWHOrYyaw7TD9ZuP4hI17PEcfhszDTBYbe8Dr%2FnDnogN9LLpDJftxxyoTbxhm86BDqVgBCfTGeq2g6hdpZsjf6CaanVSCc7BAlwg0ZMWVSEDYrrE9vg8X8pawuzy7G3ZueQ%3D%3D%20thansen%40e1405%0Assh-rsa%20AAAAB3NzaC1yc2EAAAADAQABAAABAQDVLz3tobhcdSTd4nnKvApwBKR2suEGDBfX13K2eilXm290KNGGU%2BLrAIO9b5Mm1cF27z6aZQ9Fsw5LTs4CKWTLSBV5JWtFdXyJ3B0ko8Sr6OA3Vi7QyF7076OX%2FsUTbx1egQTWsNUrknPNKwkjAD3ZMIkBgyT9L%2BrDleofjk22FTvpz40ZRNIma7P9STzvawB1OrD5d7iu0i5QY%2B2BVr2x1rugmN9GghYstTIQUl9gYnrlYmARkEdZaymc3pU5HdFNjPO%2FwPNSmZRcvUeXw6ZqhyrzaDzaQrC6pFevjLSzSpGQfFSjPOcYm9iGq5rymVDddt1%2BK8UPjwHS%2BJmrcRu1%0A

I create pre-baked images (based on the upstream provided cloud images from ubuntu, centos, etc) where common tools are added such as:

• qemu-guest-agent
• cloud-init
• nfs-utils
• iscsi
• etc

My intent it to further add the templates to rancher-ui to provision directly from there.

Is there any interest in supporting the above flow and/or would it be accepted if I hacked something together?

[lnxbil]

Plausible idea. In principle, everything is easy and possible, yet I do not have time to add features right now, there are still outstanding bugs that had to be taken care of. If you want to start hacking, just go ahead. Patches welcome.

[travisghansen]

K, I’ll start putting it together. I have some decent scripts that could be shared for creating the base images etc as well if people are interested.

I’m pretty new to machine, is my understanding correct that machine itself invokes the install url after the driver has provisioned a node and produced ssh access details?

[lnxbil]

I’m pretty new to machine, is my understanding correct that machine itself invokes the install url after the driver has provisioned a node and produced ssh access details?

Yes, docker-machine takes care of everything as long as it can access the VM via SSH. That is the crucial part of the driver and the main point that always fails (at least while I developed this driver).

[travisghansen]

Ok, but it always tries to install docker via exec over ssh the given url right? Are there scenarios where that can be disabled (ie: machine has docker baked in)?

Also from what I can tell machine essentially assumes a newly minted keypair for each node?

Thanks!

[lnxbil]

Ok, but it always tries to install docker via exec over ssh the given url right? Are there scenarios where that can be disabled (ie: machine has docker baked in)?

I haven't checked, but I think it has to be installed and it just uses it.

Also from what I can tell machine essentially assumes a newly minted keypair for each node?

IIRC, it generates a new one, yes.

[travisghansen]

Ok, I have a working prototype. Would you prefer to introduce some sort of strategy param to determine the path, or just detect the presence of a template vmid to alter behavior?

[lnxbil]

Strategy sounds good. Please also include an example in the README.md on how to use this. Looking forward to it. Sounds very interesting.

[travisghansen]

Yeah, working on a better base image ATM to do some final testing with. Spent way too much time on properly handling the sshkeys business due to this craziness: https://forum.proxmox.com/threads/how-to-use-pvesh-set-vms-sshkeys.52570/

I got it all sorted (I think) but I'm doing some crazyiness to make it work (NOTE: I know nothing about go, so I'm just fumbling along).

If you know something better than this let me know:

// NodesNodeQemuVMIDConfigSetSSHKeys access the API
// Set config options
// https://forum.proxmox.com/threads/how-to-use-pvesh-set-vms-sshkeys.52570/
// cray encoding style *AND* double-encoded
func (p ProxmoxVE) NodesNodeQemuVMIDConfigSetSSHKeys(node string, vmid string, SSHKeys string) (taskid string, err error) {
	r := strings.NewReplacer("+", "%2B", "=", "%3D", "@", "%40")

	path := fmt.Sprintf("/nodes/%s/qemu/%s/config", node, vmid)
	SSHKeys = url.PathEscape(SSHKeys)
	SSHKeys = r.Replace(SSHKeys)

	SSHKeys = url.PathEscape(SSHKeys)
	SSHKeys = r.Replace(SSHKeys)

	response, err := p.client.R().SetHeader("Content-Type", "application/x-www-form-urlencoded").SetBody("sshkeys=" + SSHKeys).Post(p.getURL(path))

	if err != nil {
		return "", err
	}
	code := response.StatusCode()

	if code < 200 || code > 300 {
		return "", fmt.Errorf("status code was '%d' and error is\n%s", code, response.Status())
	}

	var f map[string]interface{}

	err = json.Unmarshal([]byte(response.String()), &f)
	if err != nil {
		return "", err
	}
	zz, err := json.Marshal(f["data"])
	if err != nil {
		return "", err
	}

	err = json.Unmarshal(zz, &taskid)

	return taskid, err
}

As an FYI, the install script (along with several other commands) do get exec'd on the node. It just happens that if docker is already installed on the node it moves along, if not it does a whole thing.

[travisghansen]

OK, initial MVP here: #34

Basic idea is, create a template, configure it to use cloud-init (both installed in the OS image, and in proxmox). Then simply invoke the machine commands as usually giving the appropriate flags.

In my case, I had several images already that had the basic requirements (CentOS7, CentOS 8, Ubuntu 18.04, and Ubuntu 20.04) and all of them worked without alterations (to the existing images).

In short the vm is cloned, disk resized, the newly minted ssh key is added to the cloud-init data, the machine is booted (cloud-init takes care of resizing the fs to fill the disk size), and then the install url is invoked over ssh installing/configuring/etc docker.

There are some looming questions but it's a decent first stab.

[travisghansen]

Somewhat tangent, I’ve only tried the 1.5.6 release of rancher os but I don’t seem to get acpi events handled...does that work for you? For example can you use the shutdown or reboot buttons in the UI and have the machine actually do something?

After digging deeper into the code and process I would suggest removing the cdrom install use case entirely. It’s a very narrow possibility and the rancher os openstack images work fine with extremely minor changes (a cloud init file to invoke image resize and enable qemu agent). Doing so would eliminate a majority of need to tweak various machine options as they can be set on the template directly.

I have scripts that can be shared to create the custom images and/or they could simply be shared (I’ll be committing my scripts anyhow to github along with other custom proxmox utils I wrote).

[travisghansen]

OK, after dealing with rancher/os#2647 for far longer than I cared, I have a fully working solution where start/stop/restart/kill/etc all work as appropriate.

Here's the script I use to build the image to have an idea of the solution (note: I intend to clean this along with my other similar scripts for Ubuntu/CentOS before committing):

#!/bin/bash

set -x
set -e

export IMGID=9006
export IMG="rancheros-openstack.img"
export STORAGEID="bitness-nfs"

if [ ! -f "${IMG}" ];then
  wget https://github.com/rancher/os/releases/download/v1.5.6/rancheros-openstack.img
fi

if [ ! -f "${IMG}.orig" ];then
  cp -f "${IMG}" "${IMG}.orig"
fi

guestmount -a ${IMG} -m /dev/sda1 /mnt/tmp/

cat > /mnt/tmp/var/lib/rancher/conf/cloud-config.d/proxmox.yml << '__EOF__'
#cloud-config

# Giant mess of qemu-guest-agent
# https://github.com/rancher/os-services/blob/master/q/qemu-guest-agent.yml
# https://github.com/qemu/qemu/blob/master/qga/commands-posix.c#L84
# https://github.com/rancher/os/issues/2822
# https://github.com/rancher/os/issues/2647
# https://ezunix.org/index.php?title=Prevent_suspending_when_the_lid_is_closed_on_a_laptop_in_RancherOS
#
# The problem is overly complicated because of 2 things:
# - qemu-ga is hard-coded to invoke /sbin/shutdown
# - the rancher qemu-guest-agent service mounts 'volumes_from'
#    which bind mount the above path, so it's impossible to use
#    the supported image, therefor we've replaced it with a generic
#    qemu-guest-agent image
#
# Also note, due to weirdness, we simply bind mount the system-docker
# binary into the contaier and exec ros in *another* container to
# actually trigger the reboot

runcmd:
- sudo rm -rf /var/lib/rancher/resizefs.done

# when the qemu-guest-agent issue is fixed, all agent-related garbage below
# can simply be replaced by this..
#- sudo ros service enable qemu-guest-agent
#- sudo ros service up qemu-guest-agent

rancher:
  resize_device: /dev/sda

  services:
    qemu-guest-agent:
      image: linuxkit/qemu-ga:v0.8
      command: /usr/bin/qemu-ga
      privileged: true
      restart: always
      labels:
        io.rancher.os.scope: system
        io.rancher.os.after: console
      pid: host
      ipc: host
      net: host
      uts: host
      volumes:
      - /dev:/dev
      - /usr/bin/ros:/usr/bin/ros
      - /var/run:/var/run
      - /usr/bin/system-docker:/usr/bin/system-docker
      - /home/rancher/overlay/qemu-guest-agent/etc/qemu/qemu-ga.conf:/etc/qemu/qemu-ga.conf
      - /home/rancher/overlay/qemu-guest-agent/sbin/shutdown:/sbin/shutdown
      volumes_from:
      - system-volumes
      - user-volumes

write_files:

  - path: /home/rancher/overlay/qemu-guest-agent/etc/qemu/qemu-ga.conf
    permissions: "0755"
    owner: root
    content: |
      [general]
      daemon=false
      method=virtio-serial
      path=/dev/virtio-ports/org.qemu.guest_agent.0
      pidfile=/var/run/qemu-ga.pid
      statedir=/var/run
      verbose=false
      blacklist=

  - path: /home/rancher/overlay/qemu-guest-agent/sbin/shutdown
    permissions: "0755"
    owner: root
    content: |
      #!/bin/sh

      ARGS=$(echo "${@}" | sed 's/+0/now/g')
      system-docker exec console ros entrypoint shutdown $ARGS

__EOF__

umount /mnt/tmp
sync

qm create ${IMGID} --memory 512 --net0 virtio,bridge=vmbr0
qm importdisk ${IMGID} ${IMG} ${STORAGEID} --format qcow2
qm set ${IMGID} --scsihw virtio-scsi-pci --scsi0 ${STORAGEID}:${IMGID}/vm-${IMGID}-disk-0.qcow2
qm set ${IMGID} --ide2 ${STORAGEID}:cloudinit
qm set ${IMGID} --boot c --bootdisk scsi0
qm set ${IMGID} --serial0 socket --vga serial0
qm template ${IMGID}

# set host cpu, ssh key, etc
qm set ${IMGID} --scsihw virtio-scsi-pci
qm set ${IMGID} --cpu host
qm set ${IMGID} --agent enabled=1
qm set ${IMGID} --autostart 1
qm set ${IMGID} --onboot 1
qm set ${IMGID} --ostype l26
qm set ${IMGID} --ipconfig0 "ip=dhcp"

[lnxbil]

So, merged and available in v4.