snu-csl/nvmevirt

Unable to run OpenMPDK KVBench on NVMeVirt KVSSD

arjun21k opened this issue · 9 comments

arjun21k commented

Hi. I am trying to run Samsung’s OpenMPDK KVBench and sample programs (sample_code_async and sample_code_sync) on NVMeVirt’s KVSSD. I was able to configure nvmevirt to run as 10GB KVSSD on my physical machine running Ubuntu 22.04 (Ubuntu kernel v5.19.0-43-generic). Following is the output from dmesg:
image

After that I tried to use both kernel and userspace drivers from Samsung OpenMPDK/KVSSD (https://github.com/OpenMPDK/KVSSD) v1.6.0.
I use the pre-built userspace device driver (udd) found under KVSSD/PDK/core/lib/libkvnvmedd.a and compile SNIA KV API udd option. After that I try to setup SPDK environment as follows-
sudo KVSSD/PDK/core/tools/setup.sh
with following output-
image

dmesg output is as follows –
image

After that I use one thread to write 100 KV requests (8B key, 512B value) with queue depth=2 using sample_code_async binary-
sudo KVSSD/PDK/core/build/sample_code_async -d 0001:10:00.0 -n 100 -q 2 -o 1 -k 8 -v 512 -t 1
image

I get the same error when I try to use sample_code_sync program as well.
Following is dmesg log
image

Could you please help in debugging this issue?

The kernel device driver from OpenMPDK/KVSSD does not seem to support Linux kernel v5.15.x.

Could you please share the steps you followed to run KVBench with NVMeVirt’s KVSSD? Also, could you please share which driver (kernel or user-space) and its version you used while running KVBench?

Please let me know if you need any other details.

Thank you in advance.

jaehshim commented

Hello,

For the KVBench tests, we have used kernel driver and since there is no 5.x kernel driver on OpenMPDK's KVSSD repo, we have ported the 4.x kernel driver to 5.x kernel driver.
(We compared the 4.x driver to the original 4.x nvme driver and implemented the diffs to the 5.x nvme driver)

And due to some privacy issues, we did not make the code public.

arjun21k commented

Thank you for letting me know. Does NVMeVirt KVSSD work with the user-space device driver, uNVMe (https://github.com/OpenMPDK/uNVMe), in the OpenMPDK's KVSSD repo?

beowulf commented

Seems like the same issue of #14 (unable to assign MSI-X IRQ to user-space drivers).

beowulf commented

Turns out uio_pci_generic requires INTx handling, which was excluded while code cleanup. Would you like to test with e9bbeaf which reverts the removal?

arjun21k commented

Based on #14 I see that you are working on it, so I will be happy to test it out later. Also, could you please share the benchmark configuration information used for running KVBench and KVCeph on NVMeVirt in Figure 4?

jaehshim commented

We used 16B Keys for KVBench and differed the value size as shown in Figure 4.
For KVCeph, we used the rados bench and used 4K size objects.
For some reason the KVCeph repo didn't work well. So we had to use KVCeph that were inside KVSSD v0.7.1.

beowulf commented

There were the similar issues to #14 (i.e., wrong code cleaning left the side-effects on not-recently-tested modules). These are fixed and pushed under the 'kvssd' branch for testing. Please let me know if your issue persists.

arjun21k commented

Thank you. I will test it out. Also, for Figure 4, could you please also share other parameters in bench_config.ini file for KVBench like synchronous/asynchronous I/O, number of reader and writer threads, queue depth, batch distribution.

jaehshim commented

Below is our configuration for KVBench

  • Synchronous I/O (queue depth doesn't matter)
  • single thread
  • batch distribution = 1
  • 👍1