Disk hotplug fails
Closed this issue · 2 comments
I have installed last version of ganeti-rpm in centos 7.8 with kvm from scl:
[root@g1 ~]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.12.0 (qemu-kvm-ev-2.12.0-44.1.el7_8.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
[root@g1 ~]# yum info -q qemu-kvm-ev-2.12.0-44.1.el7_8.1.x86_64
Installed Packages
Name : qemu-kvm-ev
Arch : x86_64
Epoch : 10
Version : 2.12.0
Release : 44.1.el7_8.1
Size : 15 M
Repo : installed
From repo : centos-qemu-ev
Summary : QEMU is a machine emulator and virtualizer
URL : http://www.qemu.org/
License : GPLv2 and GPLv2+ and CC-BY
Description : qemu-kvm-ev is an open source virtualizer that provides hardware
: emulation for the KVM hypervisor. qemu-kvm-ev acts as a virtual
: machine monitor together with the KVM kernel modules, and emulates the
: hardware for a full system such as a PC and its associated peripherals.
Also I have installed python-fdsend with pip:
[root@g1 ~]# pip -q freeze |grep fdsend
-e git+https://github.com/fknittel/python-fdsend.git@1c8f1082d919c7e8ac97fdfd051b117790c43770#egg=fdsend
Network hotplug works without problem, but Disk hotplug it doesn't work:
[root@g1 ~]# gnt-instance modify --hotplug --disk add:size=2G testdesdeplantilla-3
Wed Oct 21 09:06:50 2020 - INFO: Trying to hotplug device...
Wed Oct 21 09:06:50 2020 - WARNING: Could not hotplug device: Error while executing backend function: kvm: error executing the device_add command: Property 'virtio-blk-device.drive' can't find value 'disk-287278b4-c873-48a8' (GenericError):
Wed Oct 21 09:06:50 2020 - INFO: Continuing execution..
Wed Oct 21 09:06:50 2020 - INFO: Waiting for instance testdesdeplantilla-3 to sync disks
Wed Oct 21 09:06:51 2020 - INFO: - device disk/1: 24.20% done, 3s remaining (estimated)
Wed Oct 21 09:06:54 2020 - INFO: Instance testdesdeplantilla-3's disks are in sync
Modified instance testdesdeplantilla-3
- disk/1 -> add:size=2048,mode=rw
- disk/1 -> hotplug:failed
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.
[root@g1 ~]#
In noded log I can see this error:
2020-10-21 09:06:50,149: ganeti-noded pid=438477 INFO RunCmd echo 'drive_add dummy file=/dev/fdset/0,if=none,id=disk-287278b4-c873-48a8,format=raw' | /usr/bin/socat STDIO UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/testdesdeplantilla-3.monitor
2020-10-21 09:06:50,158: ganeti-noded pid=438477 ERROR Error in RPC call
Traceback (most recent call last):
File "/usr/share/ganeti/2.16/ganeti/server/noded.py", line 188, in HandleRequest
result = (True, method(serializer.LoadJson(req.request_body)))
File "/usr/share/ganeti/2.16/ganeti/server/noded.py", line 659, in perspective_hotplug_device
return backend.HotplugDevice(instance, action, dev_type, device, extra, seq)
File "/usr/share/ganeti/2.16/ganeti/backend.py", line 3215, in HotplugDevice
return fn(instance, dev_type, device, extra, seq)
File "/usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/init.py", line 177, in wrapper
return fn(self, instance, *args, **kwargs)
File "/usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/init.py", line 2281, in HotAddDevice
self.qmp.HotAddDisk(device, kvm_devid, uri, drive_add_fn)
File "/usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/monitor.py", line 228, in wrapper
ret = fn(*args, **kwargs)
File "/usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/monitor.py", line 595, in HotAddDisk
self.Execute("device_add", arguments)
File "/usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/monitor.py", line 432, in Execute
ret = self._GetResponse(command)
File "/usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/monitor.py", line 459, in _GetResponse
err[self._ERROR_CLASS_KEY]))
HypervisorError: kvm: error executing the device_add command: Property 'virtio-blk-device.drive' can't find value 'disk-287278b4-c873-48a8' (GenericError):
I don't konw if it works with other version of kvm.
The hotplug is an interesting feature. I've never used it before, but when I tried it I got the same error.
Property'virtio-blk-device.drive' can't find value'disk-287278b4-c873-48a8' (GenericError) seems to be a qemu issue.
Below is an issue with a similar error. The solution may require a newer qemu.
https://bugzilla.redhat.com/show_bug.cgi?id=1662024
https://bugzilla.redhat.com/show_bug.cgi?id=1718255
I have check that if I force to use filename as it it was an userspace device, it works.
Modifiying /usr/share/ganeti/2.16/ganeti/hypervisor/hv_kvm/monitor.py in line 549:
- if os.path.exists(uri):
- if not os.path.exists(uri):
It uses link to drbd device directly instead fdset:
2020-10-22 07:25:12,806: ganeti-noded pid=266031 INFO RunCmd echo 'drive_add dummy file=/var/run/ganeti/instance-disks/test-red-int-0:4,if=none,id=disk-4105aa46-40e0-4588,format=raw' | /usr/bin/socat STDIO UNI
X-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/test-red-int-0.monitor
2020-10-22 07:25:12,818: ganeti-noded pid=266031 INFO Verifying hotplug command (retry 0): True
2020-10-22 07:25:12,819: ganeti-noded pid=266031 INFO Device disk-4105aa46-40e0-4588 has been correctly hot-plugged
I think it could be a problem with python fdsend in Centos 7 or a change in fdsets in kvm