Meet no irq handler for vector on WSL2
Closed this issue · 9 comments
I tried to use it on WSL2, linux kernel version is 5.10.16.3-microsoft-standard-WSL2, but hit interrupt drop error:
The dmesg log is like:
__common_interrupt: 4.34 No irq handler for vector
Driver can receive the first two admin command and it seems the nvme driver can't receive interrupt of nvme0q0 and report command timeout finally.
The dmesg log is like this:
[ 2499.689475] WSL2: Performing memory compaction.
[ 2501.640750] nvme nvme0: I/O 8 QID 0 timeout, disable controller
[ 2502.140780] nvme nvme0: Identify Controller failed (-4)
[ 2502.140782] nvme nvme0: Removing after probe failure status: -5
BTW, it's my .wslconfig , use 8G memory , 8cpus, leave 1G as reserved memory :
[wsl2]
memory=8GB
processors=8
kernelCommandLine = "memmap=1G$0x180000000 iommu=no-intremap intremap=off"
kernel=D:\wsl\bzImage
As a matter of fact, I have zero experience in running WSL2...
As long as it runs the Linux kernel, there will be something that corresponds to the .config of the kernel. Would you like to attach the configuration file so that I can check the interrupt handling infrastructure in WSL2? Or could you check whether APIC and MSI-X is enabled in the WSL2 kernel?
Hyper-V (and therefore WSL2) uses its own PCI controller implementation: https://github.com/microsoft/WSL2-Linux-Kernel/blob/linux-msft-wsl-5.15.y/drivers/pci/controller/pci-hyperv.c
Also, unlike QEMU, it doesn't expose a standard IBM-PC model to the guest VM, so I won't be surprised if it doesn't follow the standard APIC interrupt that x86 uses.
@hamanxu, try the below patch.
If it doesn't work, it'd be better off to either use a QEMU-backed VM or use nested virtualization (VM-inside-VM). WSL2 supports nested virtualization, so you can just use virt-manager and QEMU inside WSL2.
@beowulf, this is the WSL2 defconfig: https://github.com/microsoft/WSL2-Linux-Kernel/blob/linux-msft-wsl-5.15.y/arch/x86/configs/config-wsl
I've attached /proc/interrupts output from WSL2 below as well.
diff --git a/pci.c b/pci.c
index 6ee10970db30..949b7251d639 100644
--- a/pci.c
+++ b/pci.c
@@ -35,13 +35,9 @@ static void __init_apicid_to_cpuid(void)
static void __signal_irq(struct msi_desc *msi_desc)
{
struct irq_data *irqd = irq_get_irq_data(msi_desc->irq);
- struct irq_cfg *irqc = irqd_cfg(irqd);
+ struct irq_chip *chip = irq_data_get_irq_chip(irqd);
- unsigned int target = irqc->dest_apicid;
- unsigned int target_cpu = apicid_to_cpuid[target];
-
- NVMEV_DEBUG("vector %d, dest_apicid %d, target_cpu %d\n", irqc->vector, target, target_cpu);
- apic->send_IPI(target_cpu, irqc->vector);
+ chip->irq_retrigger(irqd);
return;
}
CPU0 CPU1 CPU2 CPU3
8: 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 0 0 0 IO-APIC 9-fasteoi acpi
24: 0 1 0 0 Hyper-V PCIe MSI 671088640-edge virtio0-config
25: 0 0 905 0 Hyper-V PCIe MSI 671088641-edge virtio0-virtqueues
26: 0 0 0 0 Hyper-V PCIe MSI 1610612736-edge virtio1-config
27: 0 179 0 0 Hyper-V PCIe MSI 1610612737-edge virtio1-requests
28: 0 0 1 0 Hyper-V PCIe MSI 0-edge virtio2-config
29: 0 0 0 1 Hyper-V PCIe MSI 1-edge virtio2-hiprio
30: 10 0 0 0 Hyper-V PCIe MSI 2-edge virtio2-requests.0
31: 0 0 0 0 Hyper-V PCIe MSI 3489660928-edge virtio3-config
32: 0 0 77 0 Hyper-V PCIe MSI 3489660929-edge virtio3-requests
33: 0 0 0 0 Hyper-V PCIe MSI 2684354560-edge virtio4-config
34: 0 0 0 0 Hyper-V PCIe MSI 2684354561-edge virtio4-requests
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 0 0 0 0 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RTR: 0 0 0 0 APIC ICR read retries
RES: 46 56 50 78 Rescheduling interrupts
CAL: 2017 2910 986 3016 Function call interrupts
TLB: 0 0 0 0 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
HYP: 3040 31 16 10 Hypervisor callback interrupts
HRE: 0 0 0 0 Hyper-V reenlightenment interrupts
HVS: 809 960 808 660 Hyper-V stimer0 interrupts
ERR: 0
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 Posted-interrupt wakeup event
@arter97 @beowulf
Thanks your patch , it works on my WSL2 enviroment and all IO commands executed normally.
It's dmesg log:
[ 15.583154] nvmev: loading out-of-tree module taints kernel.
[ 15.583809] NVMeVirt: Storage : 180100000 + 1ff00000
[ 15.584106] NVMeVirt: [NVMEV_NAMESPACE_INIT] ns=0 ns_addr=0000000063d0b537 ns_size=511(MiB)
[ 15.584107] ERR 760
[ 15.584125] PCI host bridge to bus 0001:10
[ 15.584128] pci_bus 0001:10: root bus resource [io 0x0000-0xffff]
[ 15.584129] pci_bus 0001:10: root bus resource [mem 0x00000000-0x3fffffffffff]
[ 15.584130] pci_bus 0001:10: root bus resource [bus 00-ff]
[ 15.584131] pci_bus 0001:10: scanning bus
[ 15.584136] pci 0001:10:00.0: [0c51:0101] type 00 class 0x010802
[ 15.584138] pci 0001:10:00.0: reg 0x10: [mem 0x180000000-0x180003fff 64bit]
[ 15.584140] pci 0001:10:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[ 15.584141] pci 0001:10:00.0: enabling Extended Tags
[ 15.584184] pci_bus 0001:10: fixups for bus
[ 15.584185] pci_bus 0001:10: bus scan returning with max=10
[ 15.584932] nvme 0001:10:00.0: runtime IRQ mapping not provided by arch
[ 15.585034] nvme nvme0: pci function 0001:10:00.0
[ 15.585067] NVMeVirt: Successfully created virtual PCI bus (node 0)
[ 15.585093] nvme 0001:10:00.0: enabling bus mastering
[ 15.585119] nvme 0001:10:00.0: saving config space at offset 0x0 (reading 0x1010c51)
[ 15.585120] nvme 0001:10:00.0: saving config space at offset 0x4 (reading 0x100406)
[ 15.585121] nvme 0001:10:00.0: saving config space at offset 0x8 (reading 0x1080201)
[ 15.585122] nvme 0001:10:00.0: saving config space at offset 0xc (reading 0x0)
[ 15.585123] nvme 0001:10:00.0: saving config space at offset 0x10 (reading 0x80000004)
[ 15.585123] nvme 0001:10:00.0: saving config space at offset 0x14 (reading 0x1)
[ 15.585124] nvme 0001:10:00.0: saving config space at offset 0x18 (reading 0x0)
[ 15.585125] nvme 0001:10:00.0: saving config space at offset 0x1c (reading 0x0)
[ 15.585125] nvme 0001:10:00.0: saving config space at offset 0x20 (reading 0x0)
[ 15.585126] nvme 0001:10:00.0: saving config space at offset 0x24 (reading 0x0)
[ 15.585127] nvme 0001:10:00.0: saving config space at offset 0x28 (reading 0x0)
[ 15.585127] nvme 0001:10:00.0: saving config space at offset 0x2c (reading 0x370d0c51)
[ 15.585128] nvme 0001:10:00.0: saving config space at offset 0x30 (reading 0x0)
[ 15.585129] nvme 0001:10:00.0: saving config space at offset 0x34 (reading 0x40)
[ 15.585129] nvme 0001:10:00.0: saving config space at offset 0x38 (reading 0x0)
[ 15.585130] nvme 0001:10:00.0: saving config space at offset 0x3c (reading 0x0)
[ 15.585628] NVMeVirt: nvmev_proc_io_0 started on cpu 7 (node 0)
[ 15.585732] NVMeVirt: Successfully created Virtual NVMe deivce
[ 15.585741] NVMeVirt: nvmev_dispatcher started on cpu 6 (node 0)
And nvme list show the virt nvme device:
nvme list
Node SN Model Namespace Usage Format FW Rev
/dev/nvme0n1 CSL_Virt_SN_01 CSL_Virt_MN_01 1 535.82 MB / 535.82 MB 512 B + 0 B CSL_002
@arter97 Thanks again!!
Oh, interesting.
I wasn't actually expecting it to work as iirc irq_retrigger() calls apic->send_IPI() in the end.
Do note that this method may have some performance implications as the NVMeVirt's IRQ signaler now goes through more functions and loops (irq_chip_retrigger_hierarchy).
Also note that since you're running this under WSL2, which afaik doesn't have core-to-core CPU affinity pinning functionality, you'll suffer from core migrations overhead on NVMeVirt's kthreads (which is not present from running on bare-metal machine).
Anyways, I'm glad that it worked 😃
@beowulf, let me know if you want me to post another PR for this.
I'm not sure the performance implication here is big enough to skew the benchmark data or small enough that it won't matter.
@arter97 Thanks again. So, somewhat non-standard interrupt handling in the current implementation caused the issue, right? I am not sure how much performance implication would be posed if we switch to use the full IRQ hierarchy, but supporting various environments would be good. Also, it might (hopefully...) resolve the interrupt remapping configuration issue.
How about to use the irq_retrigger() as the default but leave the current way as an alternative configuration option?
@arter97 Thanks again. So, somewhat non-standard interrupt handling in the current implementation caused the issue, right?
It seems so, but I'm not exactly sure why. I also personally don't use WSL2 for kernel development.
I am not sure how much performance implication would be posed if we switch to use the full IRQ hierarchy, but supporting various environments would be good.
I think this is pretty specific to Hyper-V.
Native bare-metal works (obviously), and QEMU also works (which is where I test NVMeVirt).
Note that this is not enough to add support for other architectures. Creating a virtual PCI bus with an arbitrary pci_sysdata is specific to x86. And even then, irq_retrigger() is an optional feature, so it's not exactly "standard" either, so that commit message can be a bit misleading :)
Also, it might (hopefully...) resolve the interrupt remapping configuration issue.
Makes sense. I'll test this out later.
@arter97 In the past, we tried to run NVMeVirt on QEMU running on a M1 Macbook, but we couldn't make that happened due to the x86-specific interrupt handling routine in the NVMeVirt. So I thought naively that this change might relieve the issue allowing NVMeVirt to run on ARM. But your comment is right. I revised the commit log to not mislead that. Much appreciated for the insightful comments!
@beowulf Well, it seems like interrupt remapping is still unsupported.
It no longer panics though (probably with recent memory fixes), NVMe just fails to probe from __pci_enable_msix() returning -EINVAL.
[ 115.287303] nvmev: loading out-of-tree module taints kernel.
[ 115.287586] NVMeVirt: Storage : 100100000 + fa000000
[ 115.289186] NVMeVirt: [NVMEV_NAMESPACE_INIT] ns=0 ns_addr=000000003243c2ed ns_size=4000(MiB)
[ 115.289227] PCI host bridge to bus 0001:10
[ 115.289229] pci_bus 0001:10: root bus resource [io 0x0000-0xffff]
[ 115.289231] pci_bus 0001:10: root bus resource [mem 0x00000000-0x7fffffffff]
[ 115.289232] pci_bus 0001:10: root bus resource [bus 00-ff]
[ 115.289237] pci 0001:10:00.0: [0c51:0101] type 00 class 0x010802
[ 115.289239] pci 0001:10:00.0: reg 0x10: [mem 0x100000000-0x100003fff 64bit]
[ 115.289241] pci 0001:10:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[ 115.289243] pci 0001:10:00.0: enabling Extended Tags
[ 115.290183] nvme nvme1: pci function 0001:10:00.0
[ 115.290248] NVMeVirt: Successfully created virtual PCI bus (node 0)
[ 115.290305] ------------[ cut here ]------------
[ 115.290306] WARNING: CPU: 1 PID: 107 at pci_msi_setup_msi_irqs+0x1b/0x90
[ 115.290310] Modules linked in: nvmev(O+) ccm hid_apple rfcomm cmac usbhid apple_mfi_fastcharge r8152 mii bnep uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 btusb videobuf2_common btrtl btmtk snd_ctl_led btintel btbcm videodev snd_soc_skl_hda_dsp bluetooth mc snd_soc_hdac_hdmi snd_soc_intel_hda_dsp_common snd_sof_probes ecdh_generic ecc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock vmw_vmci nvme_fabrics joydev sunrpc snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common snd_sof_intel_hda soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils hid_multitouch snd_soc_hdac_hda hid_generic snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus mei_pxp mei_hdcp
[ 115.290368] pmt_telemetry snd_soc_core think_lmi pmt_class snd_compress intel_rapl_msr firmware_attributes_class wmi_bmof ac97_bus snd_pcm_dmaengine i915 intel_tcc_cooling intel_powerclamp coretemp crct10dif_pclmul polyval_clmulni snd_hda_intel intel_gtt polyval_generic drm_buddy snd_intel_dspcfg ghash_clmulni_intel snd_intel_sdw_acpi ttm snd_hda_codec iwlmvm drm_display_helper snd_hda_core kvm_intel snd_hwdep cec mac80211 rapl rc_core processor_thermal_device_pci_legacy libarc4 input_leds psmouse intel_lpss_pci iwlwifi i2c_i801 thinkpad_acpi drm_kms_helper processor_thermal_device mei_me spi_intel_pci processor_thermal_rfim sysimgblt intel_lpss intel_cstate serio_raw snd_pcm i2c_smbus spi_intel nvram processor_thermal_mbox syscopyarea ledtrig_audio processor_thermal_rapl mei platform_profile idma64 sysfillrect cfg80211 intel_rapl_common fb_sys_fops snd_seq_midi intel_soc_dts_iosf thunderbolt intel_vsec xhci_pci snd_seq_midi_event igen6_edac xhci_pci_renesas ucsi_acpi typec_ucsi
[ 115.290444] snd_rawmidi typec mac_hid snd_seq i2c_hid_acpi i2c_hid snd_seq_device snd_timer video snd soundcore int3403_thermal soc_button_array int340x_thermal_zone wmi intel_hid int3400_thermal sparse_keymap pinctrl_tigerlake acpi_thermal_rel acpi_pad acpi_tad tpm_crb ramoops sch_fq_codel reed_solomon efi_pstore ip_tables x_tables autofs4
[ 115.290454] CPU: 1 PID: 107 Comm: kworker/u16:3 Tainted: G U O 6.1.23+ #1
[ 115.290455] Hardware name: LENOVO 20XWCTO1WW/20XWCTO1WW, BIOS N32ET83W (1.59 ) 02/09/2023
[ 115.290456] Workqueue: nvme-reset-wq nvme_reset_work
[ 115.290458] RIP: 0010:pci_msi_setup_msi_irqs+0x1b/0x90
[ 115.290460] Code: 05 e9 d9 d7 ff ff c3 00 00 cc cc 00 00 cc cc 55 41 57 41 56 41 54 53 4c 8b b7 f0 02 00 00 4d 85 f6 74 07 41 f6 46 28 01 75 14 <0f> 0b 41 bf ed ff ff ff 44 89 f8 5b 41 5c 41 5e 41 5f 5d c3 89 f5
[ 115.290461] RSP: 0018:ffff888207c13bd8 EFLAGS: 00010246
[ 115.290462] RAX: 0000000000000000 RBX: ffff88824d608000 RCX: 0000000000000000
[ 115.290463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88824d608000
[ 115.290464] RBP: 0000000000000001 R08: 0000000000000000 R09: ffff88825c9bae80
[ 115.290464] R10: 0000000000000000 R11: ffffc90001c06000 R12: 0000000000000001
[ 115.290465] R13: 0000000000000080 R14: 0000000000000000 R15: 0000000000000010
[ 115.290465] FS: 0000000000000000(0000) GS:ffff888843640000(0000) knlGS:0000000000000000
[ 115.290467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 115.290467] CR2: 000055af7dae8098 CR3: 0000000208012004 CR4: 0000000000770ee0
[ 115.290468] PKRU: 55555554
[ 115.290469] Call Trace:
[ 115.290470] <TASK>
[ 115.290470] ? __pci_enable_msix+0x4c9/0x6f0
[ 115.290472] ? pci_alloc_irq_vectors_affinity+0x1d5/0x210
[ 115.290474] ? nvme_reset_work+0x23e/0x1df0
[ 115.290474] ? sched_clock_cpu+0xc/0x160
[ 115.290477] ? migrate_task_rq_fair+0xa3/0x1b0
[ 115.290478] ? set_task_cpu+0x32/0x140
[ 115.290479] ? sched_clock_cpu+0xc/0x160
[ 115.290480] ? try_to_wake_up+0x417/0x580
[ 115.290481] ? process_one_work+0x1c4/0x360
[ 115.290482] ? worker_thread+0x2ed/0x6e0
[ 115.290483] ? kthread+0x215/0x250
[ 115.290485] ? rcu_free_pool+0x30/0x30
[ 115.290486] ? kthreadd+0x330/0x330
[ 115.290487] ? ret_from_fork+0x1f/0x30
[ 115.290488] </TASK>
[ 115.290489] ---[ end trace 0000000000000000 ]---
[ 115.290496] ------------[ cut here ]------------
[ 115.290496] WARNING: CPU: 1 PID: 107 at pci_msi_teardown_msi_irqs+0x1d/0x40
[ 115.290498] Modules linked in: nvmev(O+) ccm hid_apple rfcomm cmac usbhid apple_mfi_fastcharge r8152 mii bnep uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 btusb videobuf2_common btrtl btmtk snd_ctl_led btintel btbcm videodev snd_soc_skl_hda_dsp bluetooth mc snd_soc_hdac_hdmi snd_soc_intel_hda_dsp_common snd_sof_probes ecdh_generic ecc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock vmw_vmci nvme_fabrics joydev sunrpc snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common snd_sof_intel_hda soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils hid_multitouch snd_soc_hdac_hda hid_generic snd_hda_ext_core
[ 115.290517] NVMeVirt: nvmev_io_worker_0 started on cpu 7 (node 0)
[ 115.290518] snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus mei_pxp mei_hdcp pmt_telemetry snd_soc_core think_lmi pmt_class snd_compress intel_rapl_msr firmware_attributes_class wmi_bmof ac97_bus snd_pcm_dmaengine i915 intel_tcc_cooling intel_powerclamp coretemp crct10dif_pclmul polyval_clmulni snd_hda_intel intel_gtt polyval_generic drm_buddy snd_intel_dspcfg ghash_clmulni_intel snd_intel_sdw_acpi ttm snd_hda_codec iwlmvm drm_display_helper snd_hda_core kvm_intel snd_hwdep
[ 115.290529] NVMeVirt: Successfully created Virtual NVMe deivce
[ 115.290530] cec mac80211 rapl rc_core processor_thermal_device_pci_legacy libarc4 input_leds psmouse intel_lpss_pci iwlwifi i2c_i801 thinkpad_acpi drm_kms_helper processor_thermal_device mei_me spi_intel_pci processor_thermal_rfim sysimgblt intel_lpss intel_cstate serio_raw snd_pcm i2c_smbus spi_intel nvram processor_thermal_mbox syscopyarea ledtrig_audio processor_thermal_rapl mei platform_profile idma64 sysfillrect cfg80211 intel_rapl_common fb_sys_fops snd_seq_midi intel_soc_dts_iosf thunderbolt intel_vsec xhci_pci snd_seq_midi_event igen6_edac xhci_pci_renesas ucsi_acpi typec_ucsi snd_rawmidi typec mac_hid snd_seq i2c_hid_acpi i2c_hid snd_seq_device snd_timer video snd soundcore int3403_thermal soc_button_array int340x_thermal_zone wmi intel_hid int3400_thermal sparse_keymap pinctrl_tigerlake acpi_thermal_rel acpi_pad acpi_tad tpm_crb ramoops sch_fq_codel reed_solomon efi_pstore ip_tables x_tables autofs4
[ 115.290549] CPU: 1 PID: 107 Comm: kworker/u16:3 Tainted: G U W O 6.1.23+ #1
[ 115.290550] Hardware name: LENOVO 20XWCTO1WW/20XWCTO1WW, BIOS N32ET83W (1.59 ) 02/09/2023
[ 115.290551] Workqueue: nvme-reset-wq nvme_reset_work
[ 115.290552] RIP: 0010:pci_msi_teardown_msi_irqs+0x1d/0x40
[ 115.290554] Code: 00 cc cc 00 00 cc cc 00 00 cc cc 00 00 53 48 89 fb 48 8b bf f0 02 00 00 48 81 c3 d0 00 00 00 48 85 ff 74 06 f6 47 28 01 75 04 <0f> 0b eb 08 48 89 de e8 37 3d 8f ff 48 89 df 31 f6 31 d2 b9 ff ff
[ 115.290556] RSP: 0018:ffff888207c13bf8 EFLAGS: 00010246
[ 115.290557] RAX: 00000000ffffffed RBX: ffff88824d6080d0 RCX: 0000000000000000
[ 115.290563] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[ 115.290564] RBP: 0000000000000001 R08: 0000000000000000 R09: ffff88825c9bae80
[ 115.290565] R10: 0000000000000000 R11: ffffc90001c06000 R12: 0000000000000001
[ 115.290566] R13: 00000000ffffffed R14: 0000000000000000 R15: ffff88824d6080d0
[ 115.290566] FS: 0000000000000000(0000) GS:ffff888843640000(0000) knlGS:0000000000000000
[ 115.290567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 115.290568] CR2: 000055af7dae8098 CR3: 0000000208012004 CR4: 0000000000770ee0
[ 115.290569] PKRU: 55555554
[ 115.290569] Call Trace:
[ 115.290570] <TASK>
[ 115.290570] ? __pci_enable_msix+0x57c/0x6f0
[ 115.290572] ? pci_alloc_irq_vectors_affinity+0x1d5/0x210
[ 115.290580] ? nvme_reset_work+0x23e/0x1df0
[ 115.290581] ? sched_clock_cpu+0xc/0x160
[ 115.290583] ? migrate_task_rq_fair+0xa3/0x1b0
[ 115.290585] ? set_task_cpu+0x32/0x140
[ 115.290586] ? sched_clock_cpu+0xc/0x160
[ 115.290587] ? try_to_wake_up+0x417/0x580
[ 115.290588] ? process_one_work+0x1c4/0x360
[ 115.290590] ? worker_thread+0x2ed/0x6e0
[ 115.290591] ? kthread+0x215/0x250
[ 115.290592] ? rcu_free_pool+0x30/0x30
[ 115.290593] ? kthreadd+0x330/0x330
[ 115.290594] ? ret_from_fork+0x1f/0x30
[ 115.290595] </TASK>
[ 115.290595] ---[ end trace 0000000000000000 ]---
[ 115.290602] nvme nvme1: Removing after probe failure status: -22
[ 115.304656] NVMeVirt: nvmev_dispatcher started on cpu 6 (node 0)