libvfio-user

vfio-user is a framework that allows implementing PCI devices in userspace. Clients (such as qemu) talk the vfio-user protocol over a UNIX socket to a server. This library, libvfio-user, provides an API for implementing such servers.

VFIO is a kernel facility for providing secure access to PCI devices in userspace (including pass-through to a VM). With vfio-user, instead of talking to the kernel, all interactions are done in userspace, without requiring any kernel component; the kernel VFIO implementation is not used at all for a vfio-user device.

Put another way, vfio-user is to VFIO as vhost-user is to vhost.

The vfio-user protocol is intentionally modelled after the VFIO ioctl() interface, and shares many of its definitions. However, there is not an exact equivalence: for example, IOMMU groups are not represented in vfio-user.

There many different purposes you might put this library to, such as prototyping novel devices, testing frameworks, implementing alternatives to qemu's device emulation, adapting a device class to work over a network, etc.

The library abstracts most of the complexity around representing the device. Applications using libvfio-user provide a description of the device (eg. region and IRQ information) and as set of callbacks which are invoked by libvfio-user when those regions are accessed.

Memory Mapping the Device

The device driver can allow parts of the virtual device to be memory mapped by the virtual machine (e.g. the PCI BARs). The business logic needs to implement the mmap callback and reply to the request passing the memory address whose backing pages are then used to satisfy the original mmap call; more details here.

Interrupts

Interrupts are implemented via eventfd's passed from the client and registered with the library. libvfio-user consumers can then trigger interrupts by writing to the eventfd.

Building libvfio-user

Build requirements:

meson (v0.53.0 or above)
apt install libjson-c-dev libcmocka-dev or
yum install json-c-devel libcmocka-devel

The kernel headers are necessary because VFIO structs and defines are reused.

To build:

meson build
ninja -C build

Finally build your program and link with libvfio-user.so.

Supported features

With the client support found in cloud-hypervisor or the in-development qemu support, most guest VM use cases will work. See below for some details on how to try this out.

However, guests with an IOMMU (vIOMMU) will not currently work: the number of DMA regions is strictly limited, and there are also issues with some server implementations such as SPDK's virtual NVMe controller.

Currently, libvfio-user has explicit support for PCI devices only. In addition, only PCI endpoints are supported (no bridges etc.).

API

The API is currently documented via the libvfio-user header file, along with some additional documentation.

The library (and the protocol) are actively under development, and should not yet be considered a stable API or interface.

The API is not thread safe, but individual vfu_ctx_t handles can be used separately by each thread: that is, there is no global library state.

Mailing List & Chat

libvfio-user development is discussed in libvfio-user-devel@nongnu.org. Subscribe here: https://lists.gnu.org/mailman/listinfo/libvfio-user-devel.

We are on Slack at libvfio-user.slack.com (invite link); or IRC at #qemu on OFTC.

Contributing

Contributions are welcome; please file an issue or open a PR. Anything substantial is worth discussing with us first.

Please make sure to mark any commits with Signed-off-by (git commit -s), which signals agreement with the Developer Certificate of Origin v1.1.

Running make pre-push will do the same checks as done in github CI. After merging, a Coverity scan is also done.

See Testing for details on how the library is tested.

Examples

The samples directory contains various libvfio-user examples.

lspci

lspci implements an example of how to dump the PCI header of a libvfio-user device and examine it with lspci(8):

# lspci -vv -F <(build/samples/lspci)
00:00.0 Non-VGA unclassified device: Device 0000:0000
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Region 0: I/O ports at <unassigned> [disabled]
        Region 1: I/O ports at <unassigned> [disabled]
        Region 2: I/O ports at <unassigned> [disabled]
        Region 3: I/O ports at <unassigned> [disabled]
        Region 4: I/O ports at <unassigned> [disabled]
        Region 5: I/O ports at <unassigned> [disabled]
        Capabilities: [40] Power Management version 0
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

The above sample implements a very simple PCI device that supports the Power Management PCI capability. The sample can be trivially modified to change the PCI configuration space header and add more PCI capabilities.

Client/Server Implementation

Client/server implements a basic client/server model where basic tasks are performed.

The server implements a device that can be programmed to trigger interrupts (INTx) to the client. This is done by writing the desired time in seconds since Epoch to BAR0. The server then triggers an eventfd-based IRQ and then a message-based one (in order to demonstrate how it's done when passing of file descriptors isn't possible/desirable). The device also works as memory storage: BAR1 can be freely written to/read from by the host.

Since this is a completely made up device, there's no kernel driver (yet). Client implements a client that knows how to drive this particular device (that would normally be QEMU + guest VM + kernel driver).

The client exercises all commands in the vfio-user protocol, and then proceeds to perform live migration. The client spawns the destination server (this would be normally done by libvirt) and then migrates the device state, before switching entirely to the destination server. We re-use the source client instead of spawning a destination one as this is something libvirt/QEMU would normally do.

To spice things up, the client programs the source server to trigger an interrupt and then migrates to the destination server; the programmed interrupt is delivered by the destination server. Also, while the device is being live migrated, the client spawns a thread that constantly writes to BAR1 in a tight loop. This thread emulates the guest VM accessing the device while the main thread (what would normally be QEMU) is driving the migration.

Start the source server as follows (pick whatever you like for /tmp/vfio-user.sock):

rm -f /tmp/vfio-user.sock* ; build/samples/server -v /tmp/vfio-user.sock

And then the client:

build/samples/client /tmp/vfio-user.sock

After a couple of seconds the client will start live migration. The source server will exit and the destination server will start, watch the client terminal for destination server messages.

gpio

A gpio server implements a very simple GPIO device that can be used with a Linux VM.

Start the gpio server process:

rm /tmp/vfio-user.sock
./build/samples/gpio-pci-idio-16 -v /tmp/vfio-user.sock &

Next, build qemu and start a VM, as described below.

Log in to your guest VM. You'll probably need to build the gpio-pci-idio-16 kernel module yourself - it's part of the standard Linux kernel, but not usually built and shipped on x86.

Once built, you should be able to load the module and observe the emulated GPIO device's pins:

insmod gpio-pci-idio-16.ko
cat /sys/class/gpio/gpiochip480/base > /sys/class/gpio/export
for ((i=0;i<12;i++)); do cat /sys/class/gpio/OUT0/value; done

shadow_ioeventfd_server

shadow_ioeventfd_server.c and shadow_ioeventfd_speed_test.c are used to demonstrate the benefits of shadow ioeventfd, see ioregionfd for more information.

Other usage notes

qemu

vfio-user client support is not yet merged into qemu. Instead, download and build this branch of qemu.

Create a Linux install image, or use a pre-made one.

Then, presuming you have a libvfio-user server listening on the UNIX socket /tmp/vfio-user.sock, you can start your guest VM with something like this:

./x86_64-softmmu/qemu-system-x86_64 -mem-prealloc -m 256 \
-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/gpio,share=yes,size=256M \
-numa node,memdev=ram-node0 \
-kernel ~/vmlinuz -initrd ~/initrd -nographic \
-append "console=ttyS0 root=/dev/sda1 single" \
-hda ~/bionic-server-cloudimg-amd64-0.raw \
-device vfio-user-pci,socket=/tmp/vfio-user.sock

SPDK

SPDK uses libvfio-user to implement a virtual NVMe controller: see docs/spdk.md for more details.

libvirt

You can configure vfio-user devices in a libvirt domain configuration:

Add xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' to the domain element.
Enable sharing of the guest's RAM:

<memoryBacking>
  <source type='file'/>
  <access mode='shared'/>
</memoryBacking>

Pass the vfio-user device:

<qemu:commandline>
  <qemu:arg value='-device'/>
  <qemu:arg value='vfio-user-pci,socket=/var/run/vfio-user.sock,x-enable-migration=on'/>
</qemu:commandline>

History

This project was formerly known as "muser", short for "Mediated Userspace Device". It implemented a proof-of-concept VFIO mediated device in userspace. Normally, VFIO mdev devices require a kernel module; muser implemented a small kernel module that forwarded onto userspace. The old kernel-module-based implementation can be found in the kmod branch.

vmuxIO/libvfio-user