Object is remote error when using virtiofsd (fixed with --force-9p)
Closed this issue · 16 comments
Hi @arighi , have you seem this error?
$ vng --rwdir . --memory 8G
_ _
__ ()_ | | _ __ ___ ___ _ __ __ _
\ \ / / | | | _ _ \ / _ ____| _ \ / _ |
\ V /| | | | || | | | | | /| | | | (| |
_/ ||| _|| || |_|_| || ||__ |
|___/
kernel version: 6.7.0-rc3+ x86_64
mpdesouza@virtme-ng:~/git/linux> make
UPD include/generated/compile.h
xargs: rm: Object is remote
make[1]: *** [/home/mpdesouza/git/linux/Makefile:1202: remove-stale-files] Error 126
make: *** [Makefile:234: __sub-make] Error 2
I'm seem this problem since last week, but I don't quite remember when it started to happen...
There are also other issues to mount other overlay dirs:
[ 0.302116] virtme-ng-init: mount devtmpfs -> /dev: EBUSY: Device or resource busy
[ 0.303059] overlay: filesystem on /opt not supported
[ 0.303087] virtme-ng-init: mount virtme_rw_overlay2 -> /opt: EINVAL: Invalid argument
[ 0.303149] overlay: filesystem on /srv not supported
[ 0.303161] virtme-ng-init: mount virtme_rw_overlay3 -> /srv: EINVAL: Invalid argument
[ 0.303465] overlay: filesystem on /var not supported
[ 0.303481] virtme-ng-init: mount virtme_rw_overlay5 -> /var: EINVAL: Invalid argument
But again, I'm not sure what can be the problem. virtiofsd maybe? When I use --force-9p it works. Have you seen it in your setup? As I never played with virtiofsd I'm not sure what's the issue... My virtiofsd version is 1.7.2
Thanks again for working on virtme-ng!
@marcosps it looks like something related to overlayfs on top of virtiofs, I've never seen this issue before, but it reminds me a bit of https://gitlab.com/virtio-fs/virtiofsd/-/commit/4add878112adba5d105fa45df5b4522410be52c7 (that was fixed in virtiofsd-1.7.0 so you should be fine).
It might be still related to some permission restrictions on the host that are preventing virtiofsd to access all the backend files "properly". Do you have any special security/hardening features enabled on the host (i.e., selinux, apparmor or similar)?
As a very simple test maybe you could try to run vng as root and see if you still get the same errors.
So, I already had apparmor disabled, and when trying to run vng with root resulted:
qemu-system-x86_64: -chardev socket,id=charvirtfs5,path=/tmp/virtme_hstbp1h: Failed to connect to '/tmp/virtme_hstbp1h': Connection refusedSo I need to investigate it, but so far no luck to discover what's wrong... =/
Hi @marcosps, how can I try to reproduce this issue locally? I guess you're testing this in SuSE / openSuSE, can I just install the latest version and run vng on it?
Hi @marcosps, how can I try to reproduce this issue locally? I guess you're testing this in SuSE / openSuSE, can I just install the latest version and run vng on it?
Sorry for the late reply! Yes, I'm testing my openSUSE Tumbleweed machine. I'll make a fresh install and try again, since I can messed things up while trying to fix some stuff in the past :P
Hey @marcosps I still need to look at this sorry... quick question, is CONFIG_OVERLAY_FS set in the kenrel? Is it set to =m?
Asking because I've seen a similar issue yesterday and the problem was that overlayfs was disabled in the kernel. And I was wondering, if it's set to m maybe for some reasons it's failing to detect and load the module?
Hey @marcosps I still need to look at this sorry... quick question, is
CONFIG_OVERLAY_FSset in the kenrel? Is it set to=m?Asking because I've seen a similar issue yesterday and the problem was that overlayfs was disabled in the kernel. And I was wondering, if it's set to
mmaybe for some reasons it's failing to detect and load the module?
Hi @arighi ! NP, I didn't have time to dig into this deeper myself =/
But currently I have OVERLAY_FS set as builtin, as you can see here:
mpdesouza@virtme-ng:~/git/linux> zcat /proc/config.gz | grep CONFIG_OVERLAY_FS=
CONFIG_OVERLAY_FS=y
mpdesouza@virtme-ng:~/git/linux> make
xargs: rm: Object is remote
make[1]: *** [/home/mpdesouza/git/linux/Makefile:1202: remove-stale-files] Error 126
make: *** [Makefile:240: __sub-make] Error 2
So unfortunately I'm not sure what can be wrong in this case =/
Also, there are my settings for VIRTIO:
mpdesouza@virtme-ng:~/git/linux> zcat /proc/config.gz | grep VIRTIO
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
# CONFIG_HW_RANDOM_VIRTIO is not set
CONFIG_VIRTIO_ANCHOR=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_PCI_LIB_LEGACY=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_ADMIN_LEGACY=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_VIRTIO_MEM is not set
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_MMIO=y
# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set
CONFIG_VIRTIO_IOMMU=y
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=y
# CONFIG_CRYPTO_DEV_VIRTIO is not set
configs look good, ok, thanks for double checking, I'll try to reproduce this locally.
@marcosps I have some news!
I've decided to install the latest tumbleweed on my old laptop and do some tests. And I ran into a bunch of issues with vng...
First of all you need 35e954a if you want to use the host's kernel (vng -r), but this is a different issue, and it should be fixed now.
Another issue is that I need to disable microvm (--disable-microvm) or virtiofsd doesn't work. Initially I thought that it was related to a newer qemu (8.2.0 in tumbleweed), but I also tried the newer 8.2.1 from Ubuntu noble-proposed and that works... so I still don't know why microvm doesn't work with virtiofsd. But this is another issue again (and for now let's pretend that we're happy with the --disable-microvm workaround).
So, focusing at the Object is remote error. This happens with overlayfs when there is a dentry that, according to overlayfs, is "weird", see ovl_dentry_weird(). When this happens overlayfs just returns EREMOTE, skipping any operation with that dentry, hence the Object is remote error.
In our particular case it happens because DCACHE_NEED_AUTOMOUNT is set in dentry->d_flags. And this happens because of the btrfs subvol mount option.
For example (with the default /etc/fstab):
$ grep local /etc/fstab
UUID=1363cb98-f8b0-46b2-bea7-4e5131cd47a4 /usr/local btrfs subvol=/@/usr/local 0 0
$ ~/src/virtme-ng/vng --disable-microvm -- stat /usr/local
stat: cannot statx '/usr/local': Object is remote
If I comment out the /usr/local the entry in /etc/fstab (and reboot):
$ grep local /etc/fstab
#UUID=1363cb98-f8b0-46b2-bea7-4e5131cd47a4 /usr/local btrfs subvol=/@/usr/local 0 0
$ ~/src/virtme-ng/vng --disable-microvm -- stat /usr/local
File: /usr/local
Size: 80 Blocks: 0 IO Block: 4096 directory
Device: 0,37 Inode: 10 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2024-02-15 16:21:17.893790974 +0100
Modify: 2024-02-15 16:20:27.043790971 +0100
Change: 2024-02-15 16:20:27.043790971 +0100
Birth: -
So, at least we know the problem now and it really seems to be the same as yours. I bet if you comment out your subvol entries in fstab you shouldn't see the problem.
How do we fix it?
I'm not sure we can do very much in virtme-ng, honestly it seems more like an overlayfs/virtiofsd interaction issue. In fact, without addition virtiofsd to the equation the problem doesn't happen. I still don't know understand how the DCACHE_NEED_AUTOMOUNT flag is set when virtiofsfd is joining the game, if we figure this out we may fix this in virtiofsd and be happy.
In the meantime I think the best for virtme-ng is to add a not to the troubleshooting section.
Opinions?
Alright, I think I may have actually found a proper fix, virtiofsd --no-announce-submounts is the answer. I'll push a new fix.
@marcosps fix pushed to main, if you can do a quick test it'd be great, but I'm pretty sure it fixes this issue.
Great, that worked, thanks a lot! It worked for me! Thanks a lot for fixing it for me!
(Now I found an issue when installing the vng from main branch, which makes the bash to be stuck, unless I specify --no-virtme-ng-init, but that's an issue for another day)
Fixed! Closing!
Great, that worked, thanks a lot! It worked for me! Thanks a lot for fixing it for me!
(Now I found an issue when installing the vng from main branch, which makes the bash to be stuck, unless I specify --no-virtme-ng-init, but that's an issue for another day)
Thanks for testing! And let's open another issue for the bash one! :)
Great, that worked, thanks a lot! It worked for me! Thanks a lot for fixing it for me!
(Now I found an issue when installing the vng from main branch, which makes the bash to be stuck, unless I specify --no-virtme-ng-init, but that's an issue for another day)Thanks for testing! And let's open another issue for the bash one! :)
So I fixed the issue by compiling virtme-ng-init from virtme-ng git branch, and then run pixp install --force, and it worked! Sorry for the noise, everything is working amazingly here! Thanks a lot for your work on this tool!
I think this one solved the boot hang on Debian Unstable too. After upgraded from 1.20 to 1.21 virtme-ng booting successfully with microvm and virtiofs enabled (I used --force-9p since 1.17).