ppc64le: Hot addition vCPUs to Kata Container fails on ppc64le

Description of problem

Hot addition of vCPUs to Kata Container fails on ppc64le

Expected result

docker run  --runtime kata-runtime --rm --cgroup-parent 0 --cpus 2 debian bash
*No error*

Actual result

 docker run  --runtime kata-runtime --rm --cgroup-parent 0 --cpus 2 debian bash
docker: Error response from daemon: OCI runtime create failed: failed to hot add vCPUs: only 0 vCPUs of 2 were added: unknown.

Kata-collect-data.sh output:-

kata-collect-data-HotAdd.log

Earlier for memory and CPU hotplug to work on IBM Power Systems, we needed to install some packages like powerpc-utils, ppc64_diag and librtas in the VM. However, in the recent kernel versions, there is an in-kernel hotplug feature that supports hotplugging without installing those packages.

I am planning to bump the Kernel version to 4.19 for ppc64le. I was wondering if there is anything specific that needs to be done or enabled in the kernel config file for hotplug to work.
/ cc @jodh-intel @grahamwhaley @jcvenegas

Hi @nitkon - there might be clues in the config fragments over in the kata-containers/packaging#314 PR. A quick grep of 'PLUG' shows items in ACPI and PCI domains - but, that may be quite different for your architecture ;-) There is of course CONFIG_MEMORY_HOTPLUG.
On the general 4.19 item, also see #1111

Even after bumping to Kernel 4.19, I see the following error in the debug ...

Jan 27 20:49:57 soe13 kata-runtime[37967]: time="2019-01-27T20:49:57.269463757+05:30" level=info msg="HOTPLUGGABLE VCPUS are !(EXTRA []qemu.HotpluggableCPU=[{host-spapr-cpu-core 1 {0 0 19 0} } {host-spapr-cpu-core 1 {0 0 18 0} } {host-spapr-cpu-core 1 {0 0 17 0} } {host-spapr-cpu-core 1 {0 0 16 0} } {host-spapr-cpu-core 1 {0 0 15 0} } {host-spapr-cpu-core 1 {0 0 14 0} } {host-spapr-cpu-core 1 {0 0 13 0} } {host-spapr-cpu-core 1 {0 0 12 0} } {host-spapr-cpu-core 1 {0 0 11 0} } {host-spapr-cpu-core 1 {0 0 10 0} } {host-spapr-cpu-core 1 {0 0 9 0} } {host-spapr-cpu-core 1 {0 0 8 0} } {host-spapr-cpu-core 1 {0 0 7 0} } {host-spapr-cpu-core 1 {0 0 6 0} } {host-spapr-cpu-core 1 {0 0 5 0} } {host-spapr-cpu-core 1 {0 0 4 0} } {host-spapr-cpu-core 1 {0 0 3 0} } {host-spapr-cpu-core 1 {0 0 2 0} } {host-spapr-cpu-core 1 {0 0 1 0} } {host-spapr-cpu-core 1 {0 0 0 0} /machine/unattached/device[0]}])"

Jan 27 20:49:57 soe13 kata-runtime[37967]: time="2019-01-27T20:49:57.269563625+05:30" level=info msg="{"arguments":{"core-id":"19","driver":"host-spapr-cpu-core","id":"cpu-0","socket-id":"0","thread-id":"0"},"execute":"device_add"}" arch=ppc64le command=create container=f79375f64d7aaa0533ce62e6d2100b15727e2e55ebfc52e02423123dc8996990 name=kata-runtime pid=37967 source=virtcontainers subsystem=qmp

Jan 27 20:49:57 soe13 kata-runtime[37967]: time="2019-01-27T20:49:57.272369474+05:30" level=info msg="{"error": {"class": "GenericError", "desc": "Property '.thread-id' not found"}}" arch=ppc64le command=create container=f79375f64d7aaa0533ce62e6d2100b15727e2e55ebfc52e02423123dc8996990 name=kata-runtime pid=37967 source=virtcontainers subsystem=qmp

Probably arch specific, how are other archs doing it...

runtime/vendor/github.com/intel/govmm/qemu/qmp.go

Line 1154 in 6f2c036

"thread-id": threadID,

So, if we want to handle qmp calls differently based on which arch we are running Kata on, where do we patch? I don't think that we can make arch specific changes to intel/govmm/qemu/qmp.go

Example: We do not provide thread-id when hotplugging on ppc64le.

Hi @nitkon - my guess would have been govmm, but it probably depends on if the specific option is wired into govmm, or if we pass it into the govmm call....?

/cc @markdryan

We do already have some architecture specific code in govmm for s390x. If there's no other way to solve the issue we could introduce a ppc64Ie version of ExecuteCPUDeviceAdd. How would it differ from the existing implementation? Would it simply ignore the threadid?

@markdryan : Works on ppc64le if we do not pass threadID and socketID.

# docker run -it  --runtime kata-runtime --cpus 3 fedora bash
[root@ee74bd6ad33f /]# lscpu
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           4
NUMA node(s):        1
Model:               2.1 (pvr 004b 0201)
Model name:          POWER8 (architected), altivec supported
L1d cache:           64K
L1i cache:           32K
NUMA node0 CPU(s):   0-3

I will send a patch soon where we need not make arch specific changes to govmm.