nvme:nvmeCoreLogError:333
kdulep opened this issue · 25 comments
nvme-1.2.0.27-4vmw.550.0.0.1331820.x86_64.vib
6.0 U2
HP Microserver Gen8
INTEL NVME P3600
dmesg:
2016-05-22T10:07:33.017Z cpu0:35374)nvme:nvmeCoreLogError:333:command failed: 0x4302fea0d040.
2016-05-22T10:09:44.427Z cpu0:33196)nvme:nvmeCoreLogError:333:command failed: 0x4302fea0d540.
2016-05-22T10:11:45.263Z cpu0:32781)nvme:nvmeCoreLogError:333:command failed: 0x4302fea0da40.
2016-05-22T10:13:46.020Z cpu0:35374)nvme:nvmeCoreLogError:333:command failed: 0x4302fea0df40.
2016-05-22T10:15:46.521Z cpu0:35374)nvme:nvmeCoreLogError:333:command failed: 0x4302fea0e440.
After removing partitions and detach:
2016-05-22T10:38:56.275Z cpu1:34322 opID=6d0c3e71)ScsiDevice: 1741: Device t10.NVMe____INTEL_SSDPEDME400G4_____________________CVMD517401J4400AGN__00000001 has been turned off administratively.
2016-05-22T10:39:57.151Z cpu0:35374)nvme:nvmeCoreLogError:333:command failed: 0x4302fea12180.
2016-05-22T10:41:58.321Z cpu0:33198)nvme:nvmeCoreLogError:333:command failed: 0x4302fea12680.
2016-05-22T10:43:59.370Z cpu0:32781)nvme:nvmeCoreLogError:333:command failed: 0x4302fea12b80.
2016-05-22T10:45:59.979Z cpu0:35374)nvme:nvmeCoreLogError:333:command failed: 0x4302fea13080.
2016-05-22T10:48:00.980Z cpu0:39425)nvme:nvmeCoreLogError:333:command failed: 0x4302fea13580.
2016-05-22T10:50:01.576Z cpu0:39571)nvme:nvmeCoreLogError:333:command failed: 0x4302fea13a80.
2016-05-22T10:52:02.080Z cpu0:35371)nvme:nvmeCoreLogError:333:command failed: 0x4302fea13f80.
Not sure what the context is. Could you please provide a detailed description of the problem? Like how the partition and device is removed or detached?
The error message indicates that the NVMe command is returned a failed status by controller.
its spam logs with this message even no activity
when driver loaded it starts and continues
even volume unmounted and device deatached
Intel driver dont shows like this error
I suspect something wrong with your device. Do you have some management tool to get the error and SMART log page?
esxcli storage core device smart get -d t10.NVMe____INTEL_SSDPEDME400G4_____________________CVMD517401J4400AGN__00000
001
Parameter Value Threshold Worst
Health Status OK N/A N/A
Media Wearout Indicator N/A N/A N/A
Write Error Count N/A N/A N/A
Read Error Count N/A N/A N/A
Power-on Hours N/A N/A N/A
Power Cycle Count N/A N/A N/A
Reallocated Sector Count N/A N/A N/A
Raw Read Error Rate N/A N/A N/A
Drive Temperature 33 253 N/A
Driver Rated Max Temperature N/A N/A N/A
Write Sectors TOT Count N/A N/A N/A
Read Sectors TOT Count N/A N/A N/A
Initial Bad Block Count N/A N/A N/A
Do you have other management tool? Since the esxcli smart command is not able to get the media errors and number of error information log entries.
do vmware have it? shall i get nvme log in linux?
last time i get no errors in intel diagnostics tool
Well, Linux error and SMART page log is fine. Let me know if the device could work fine in Linux while it doesn't work on ESXi.
Or could you try with Intel driver on ESXi and see whether it works?
device works ok but lacks delete support and spam log its same issue
it work on windows , esxi and with vmware and intel driver
i suspect no delete support and log is spammy too
intel driver dont give any log errors
its still gives error, maybe i shall enable verbose messages in log? how do i do this
2016-05-23T08:25:02.916Z cpu0:33197)nvme:nvmeCoreLogError:333:command failed: 0x430443c69dc0.
2016-05-23T08:27:03.521Z cpu0:33196)nvme:nvmeCoreLogError:333:command failed: 0x430443c6a2c0.
2016-05-23T08:29:04.052Z cpu0:33198)nvme:nvmeCoreLogError:333:command failed: 0x430443c6a7c0.
2016-05-23T08:31:04.598Z cpu0:33198)nvme:nvmeCoreLogError:333:command failed: 0x430443c6acc0.
2016-05-23T08:33:05.090Z cpu0:33197)nvme:nvmeCoreLogError:333:command failed: 0x430443c6b1c0.
2016-05-23T08:35:05.627Z cpu0:33196)nvme:nvmeCoreLogError:333:command failed: 0x430443c6b6c0.
2016-05-23T08:37:06.166Z cpu0:33197)nvme:nvmeCoreLogError:333:command failed: 0x430443c6bbc0.
2016-05-23T08:39:06.703Z cpu0:39796)nvme:nvmeCoreLogError:333:command failed: 0x430443c6c0c0.
2016-05-23T08:41:07.234Z cpu0:35373)nvme:nvmeCoreLogError:333:command failed: 0x430443c6c5c0.
2016-05-23T08:43:07.757Z cpu0:35373)nvme:nvmeCoreLogError:333:command failed: 0x430443c6cac0.
2016-05-23T08:45:08.251Z cpu0:33198)nvme:nvmeCoreLogError:333:command failed: 0x430443c6cfc0.
2016-05-2
2016-05-23T08:47:08.874Z cpu0:35373)nvme:nvmeCoreLogError:333:command failed: 0x430443c6d4c0.
2016-05-23T08:49:09.363Z cpu0:33196)nvme:nvmeCoreLogError:333:command failed: 0x430443c6d9c0.
2016-05-23T08:51:09.954Z cpu0:35424)nvme:nvmeCoreLogError:333:command failed: 0x430443c6dec0.
2016-05-23T08:53:10.459Z cpu0:35373)nvme:nvmeCoreLogError:333:command failed: 0x430443c6e3c0.
2016-05-23T08:55:10.966Z cpu0:33198)nvme:nvmeCoreLogError:333:command failed: 0x430443c6ea00.
Firstly unload the NVMe driver. Driver could be unloaded only when there is no IO on the device. So stop all IO on NVMe device. Then unclaim the path by: esxcli storage core claiming unclaim -t driver -D nvme,
Then unload the driver by: vmkload_mod -u nvme
The process is a little bit tricky, sometimes the driver unload will fail because there are system process accessing the NVMe device.
If luckily it succeed, load the driver by: vmkload_mod nvme nvme_log_level=5 nvme_dbg=0xf
And then restart the device manager by: kill -SIGHUP pidof vmkdevmgr
Now check the kernel log, if it still error. Please provide the log.
2016-05-23T09:27:46.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:27:46.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:27:46.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:27:47.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:27:47.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:27:47.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:27:48.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:27:48.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:27:48.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:27:49.685Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:27:49.685Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:27:49.685Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:27:50.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:27:50.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:27:50.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:27:51.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:27:51.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:27:51.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:27:52.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:27:52.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:27:52.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:27:53.685Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:27:53.685Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:27:53.685Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:27:54.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:27:54.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:27:54.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:27:55.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:27:55.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:27:55.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:27:56.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:27:56.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:27:56.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:27:57.685Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:27:57.685Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:27:57.685Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:27:58.683Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:27:58.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:27:58.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:27:59.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:27:59.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:27:59.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:28:00.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:28:00.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:28:00.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:28:01.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:28:01.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:28:01.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:28:02.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:28:02.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:28:02.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:28:03.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:28:03.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:28:03.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:28:04.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:28:04.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:28:04.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:28:05.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:28:05.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:28:05.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:28:06.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:28:06.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:28:06.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:28:07.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:28:07.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:28:07.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:28:08.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:28:08.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:28:08.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:28:09.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:28:09.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:28:09.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:28:10.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:28:10.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:28:10.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:28:11.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:28:11.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:28:11.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:28:12.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:28:12.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:28:12.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:28:13.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:28:13.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:28:13.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:28:14.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:28:14.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:28:14.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:28:15.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:28:15.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:28:15.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:28:16.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:28:16.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:28:16.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:28:17.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:28:17.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:28:17.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:28:18.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:28:18.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:28:18.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:28:19.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:28:19.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:28:19.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:28:20.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:28:20.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:28:20.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:28:21.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:28:21.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:28:21.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:28:22.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:28:22.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:28:22.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:28:23.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:28:23.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:28:23.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:28:24.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:28:24.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:28:24.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:28:25.683Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:28:25.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:28:25.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:28:26.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:28:26.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:28:26.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:28:27.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:28:27.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:28:27.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:28:28.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:28:28.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:28:28.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:28:29.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:28:29.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:28:29.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:28:30.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:28:30.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:28:30.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:28:31.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:28:31.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:28:31.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:28:32.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:28:32.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:28:32.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:28:33.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:28:33.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:28:33.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:28:34.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:28:34.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:28:34.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:28:35.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:28:35.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:28:35.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:28:36.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:28:36.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:28:36.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:28:37.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:28:37.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:28:37.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:28:38.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:28:38.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:28:38.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:28:39.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:28:39.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:28:39.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:28:40.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:28:40.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:28:40.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:28:41.685Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:28:41.685Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:28:41.685Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:28:42.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:28:42.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:28:42.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:28:43.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:28:43.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:28:43.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:28:44.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:28:44.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:28:44.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:28:45.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:28:45.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:28:45.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:28:46.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:28:46.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:28:46.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:28:47.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:28:47.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:28:47.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:28:48.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:28:48.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:28:48.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:28:49.685Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:28:49.685Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:28:49.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:28:50.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:28:50.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:28:50.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:28:51.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:28:51.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:28:51.686Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:28:52.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:28:52.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:28:52.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:28:53.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:28:53.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:28:53.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:28:54.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:28:54.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:28:54.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:28:55.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:28:55.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:28:55.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:28:56.685Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:28:56.685Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:28:56.685Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:28:57.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:28:57.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:28:57.685Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:28:58.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:28:58.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:28:58.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:28:59.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:28:59.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:28:59.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:29:00.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:29:00.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:29:00.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:29:01.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:29:01.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:29:01.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:29:02.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:29:02.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:29:02.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:29:03.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:29:03.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:29:03.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:29:04.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:29:04.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:29:04.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:29:05.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:29:05.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:29:05.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:29:06.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:29:06.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:29:06.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:29:07.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:29:07.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:29:07.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:29:08.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:29:08.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:29:08.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:29:09.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:29:09.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational 2016-05-23T09:29:09.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 8 2016-05-23T09:29:10.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 8 2016-05-23T09:29:10.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 8, ctrlrState [2]: Operational 2016-05-23T09:29:10.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 9 2016-05-23T09:29:11.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 9 2016-05-23T09:29:11.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 9, ctrlrState [2]: Operational 2016-05-23T09:29:11.686Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 0 2016-05-23T09:29:12.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 0 2016-05-23T09:29:12.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 0, ctrlrState [2]: Operational 2016-05-23T09:29:12.686Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 1 2016-05-23T09:29:13.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 1 2016-05-23T09:29:13.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 1, ctrlrState [2]: Operational 2016-05-23T09:29:13.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 2 2016-05-23T09:29:14.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 2 2016-05-23T09:29:14.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 2, ctrlrState [2]: Operational 2016-05-23T09:29:14.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 3 2016-05-23T09:29:15.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 3 2016-05-23T09:29:15.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 3, ctrlrState [2]: Operational 2016-05-23T09:29:15.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 4 2016-05-23T09:29:16.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 4 2016-05-23T09:29:16.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 4, ctrlrState [2]: Operational 2016-05-23T09:29:16.684Z cpu0:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 5 2016-05-23T09:29:17.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 5 2016-05-23T09:29:17.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 5, ctrlrState [2]: Operational 2016-05-23T09:29:17.684Z cpu1:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 6 2016-05-23T09:29:18.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 6 2016-05-23T09:29:18.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 6, ctrlrState [2]: Operational 2016-05-23T09:29:18.684Z cpu2:37205)nvme:NvmeCtrlr_Timeout:2622:new timeout_id 7 2016-05-23T09:29:19.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2555:In Timer 7 2016-05-23T09:29:19.684Z cpu3:37205)nvme:NvmeCtrlr_Timeout:2618:TimeoutId 7, ctrlrState [2]: Operational
Thanks, @kdulep From the log I don't see any error message.
Looks it works fine.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeMgmt_ListAdapters:147:Adapter 0: vmhba2 is online.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 255.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [21] 0x43105f627b80.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeIo_ProcessPrps:62:length 4096, dma_addr 0x13a4c4000, offset 0x0, dma_len 4096
2016-05-23T09:33:47.985Z cpu3:33170)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [21] 0x43105f627b80 back to queue [0], nrAct: 1.
2016-05-23T09:33:47.985Z cpu3:33170)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 21, head 21.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f627b80 [21] on queue 0, wait:0x0 cmd:0.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 255.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [22] 0x43105f627cc0.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeIo_ProcessPrps:62:length 4096, dma_addr 0x13a4c4000, offset 0x0, dma_len 4096
2016-05-23T09:33:47.985Z cpu3:33170)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [22] 0x43105f627cc0 back to queue [0], nrAct: 1.
2016-05-23T09:33:47.985Z cpu3:33170)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 22, head 22.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f627cc0 [22] on queue 0, wait:0x0 cmd:0.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 0.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [23] 0x43105f627e00.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeIo_ProcessPrps:62:length 512, dma_addr 0x13a4c4000, offset 0x0, dma_len 512
2016-05-23T09:33:47.985Z cpu3:33170)nvme:NvmeCore_GetStatus:462:Command failed: 0xa, INVALID FIELD IN CDB.
2016-05-23T09:33:47.985Z cpu3:33170)nvme:nvmeCoreLogError:333:command failed: 0x43105f627e00.
2016-05-23T09:33:47.985Z cpu3:33170)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [23] 0x43105f627e00 back to queue [0], nrAct: 1.
2016-05-23T09:33:47.985Z cpu3:33170)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 23, head 23.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f627e00 [23] on queue 0, wait:0x0 cmd:0.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 0.
2016-05-23T09:33:47.985Z cpu0:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [24] 0x43105f627f40.
2016-05-23T09:33:47.985Z cpu3:33170)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [24] 0x43105f627f40 back to queue [0], nrAct: 1.
2016-05-23T09:33:47.986Z cpu3:33170)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 24, head 24.
2016-05-23T09:33:47.986Z cpu0:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f627f40 [24] on queue 0, wait:0x0 cmd:0.
2016-05-23T09:33:48.684Z cpu1:37205)nvm
2016-05-23T09:35:48.589Z cpu2:35375)nvme:NvmeMgmt_ListAdapters:147:Adapter 0: vmhba2 is online. 2016-05-23T09:35:48.589Z cpu2:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 255. 2016-05-23T09:35:48.589Z cpu2:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [25] 0x43105f628080. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeIo_ProcessPrps:62:length 4096, dma_addr 0x114072000, offset 0x0, dma_len 4096 2016-05-23T09:35:48.590Z cpu3:32892)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [25] 0x43105f628080 back to queue [0], nrAct: 1. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 25, head 25. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f628080 [25] on queue 0, wait:0x0 cmd:0. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 255. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [26] 0x43105f6281c0. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeIo_ProcessPrps:62:length 4096, dma_addr 0x114072000, offset 0x0, dma_len 4096 2016-05-23T09:35:48.590Z cpu3:32892)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [26] 0x43105f6281c0 back to queue [0], nrAct: 1. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 26, head 26. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f6281c0 [26] on queue 0, wait:0x0 cmd:0. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 0. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [27] 0x43105f628300. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeIo_ProcessPrps:62:length 512, dma_addr 0x114072000, offset 0x0, dma_len 512 2016-05-23T09:35:48.590Z cpu3:32892)nvme:NvmeCore_GetStatus:462:Command failed: 0xa, INVALID FIELD IN CDB. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:nvmeCoreLogError:333:command failed: 0x43105f628300. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [27] 0x43105f628300 back to queue [0], nrAct: 1. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 27, head 27. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f628300 [27] on queue 0, wait:0x0 cmd:0. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:kernelCbIoctl:527:Ioctl cmd 1 to ctrlr nvme00070000 ns 0. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_GetCmdInfo:715:Queue [0] Cmd Info [28] 0x43105f628440. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:NvmeCore_PutCmdInfo:685:Put Cmd Info [28] 0x43105f628440 back to queue [0], nrAct: 1. 2016-05-23T09:35:48.590Z cpu2:35375)nvme:NvmeCore_SubmitCommandWait:982:sync cmd 0x43105f628440 [28] on queue 0, wait:0x0 cmd:0. 2016-05-23T09:35:48.590Z cpu3:32892)nvme:nvmeCoreProcessCq:659:Sub Queue Entries [255] tail 28, head 28.
is it ok?
specialy
2016-05-23T09:37:49.357Z cpu3:35375)nvme:NvmeCore_GetStatus:462:Command failed: 0xa, INVALID FIELD IN CDB.
maybe this causes messages spam
Sorry, I didn't capture the error.
It looks there are some management tool which issues invalid command to driver, via kernelCbIoctl.
Do you use some kinds of these tools?
Or could you open more log swtiches by: vmkload_mod nvme nvme_log_level=5 nvme_dbg=0x7fffffff?
if HP image uses it not sure, nothing special installed
upd:
https://www.dropbox.com/s/yvogu3fpsxst30q/vmkernel.zip?dl=0
nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
temperature : 33 C
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 0%
data_units_read : 2,689,321
data_units_written : 1,875,176
host_read_commands : 41,772,961
host_write_commands : 31,652,528
controller_busy_time : 2
power_cycles : 188
power_on_hours : 6,409
unsafe_shutdowns : 102
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 0 C
Temperature Sensor 2 : 0 C
Temperature Sensor 3 : 0 C
Temperature Sensor 4 : 0 C
Temperature Sensor 5 : 0 C
Temperature Sensor 6 : 0 C
Temperature Sensor 7 : 0 C
Temperature Sensor 8 : 0 C
isdct show -all -intelssd
- Intel SSD DC P3600 Series CVMD517401J4400AGN -
AggregationThreshold : 0
AggregationTime : 0
ArbitrationBurst : 0
Bootloader : 8B1B0131
CoalescingDisable : 1
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
EndToEndDataProtCapabilities : 17
EnduranceAnalyzer : 1815.12 years
ErrorString :
Firmware : 8DV10171
FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.
HighPriorityWeightArbitration : 0
IOCompletionQueuesRequested : 30
IOSubmissionQueuesRequested : 30
Index : 0
Intel : True
IntelGen3SATA : False
IntelNVMe : True
InterruptVector : 0
LBAFormat : 0
LatencyTrackingEnabled : False
LowPriorityWeightArbitration : 0
MaximumLBA : 781422767
MediumPriorityWeightArbitration : 0
MetadataSetting : 0
ModelNumber : INTEL SSDPEDME400G4
NVME_1_0_Supported : True
NVME_1_2_Supported : False
NVMeControllerID : 0
NVMePowerState : 0
NamespaceId : 1
NativeMaxLBA : 781422767
NumErrorLogPageEntries : 63
NumLBAFormats : 6
OEM : Generic
PCILinkGenSpeed : 3
PCILinkWidth : 4
PowerGovernorMode : 0 25W
Product : Middledale
ProductFamily : Intel SSD DC P3600 Series
ProductProtocol : NVME
ProtectionInformation : 0
ProtectionInformationLocation : 0
SMARTHealthCriticalWarningsConfiguration : 0
SMBusAddress : 106
SectorSize : 512
SerialNumber : CVMD517401J4400AGN
TCGSupported : False
TempThreshold : 85
TimeLimitedErrorRecovery : 0
TrimSupported : True
VolatileWriteCacheEnabled : False
WriteAtomicityDisableNormal : 0
root@it:~# isdct show -sensor -intelssd
- Intel SSD DC P3600 Series CVMD517401J4400AGN -
AvailableSpare : 100
AverageNandEraseCycles : 9
CrcErrorCount : 0
DeviceStatus : Healthy
EndToEndErrorDetectionCount : 0
EnduranceAnalyzer : 1815.12
EraseFailCount : 0
ErrorInfoLogEntries : 0x00
HighestLifetimeTemperature : 45
LowestLifetimeTemperature : 14
MaxNandEraseCycles : 12
MediaErrors : 0x00
MinNandEraseCycles : 8
PercentageUsed : 0
PowerCycles : 0x0BC
PowerOnHours : 0x01909
ProgramFailCount : 0
SpecifiedPCBMaxOperatingTemp : 85
SpecifiedPCBMinOperatingTemp : 0
Temperature : 33 Celsius
ThermalThrottleCount : 0
ThermalThrottleStatus : 0
UnsafeShutdowns : 0x066
2016-05-23T10:10:44.028Z cpu3:35399)nvme:NvmeDebug_DumpCmd:49:00: 000b0002 00000001 00000000 00000000
2016-05-23T10:10:44.028Z cpu3:35399)nvme:NvmeDebug_DumpCmd:49:04: 00000000 00000000 0e915000 00000001
2016-05-23T10:10:44.028Z cpu3:35399)nvme:NvmeDebug_DumpCmd:49:08: 00000000 00000000 007f0002 00000000
2016-05-23T10:10:44.028Z cpu3:35399)nvme:NvmeDebug_DumpCmd:49:0c: 00000000 00000000 00000000 00000000
2016-05-23T10:10:44.028Z cpu1:33013)nvme:NvmeDebug_DumpCpl:62:00: 00000000 00000000 0000000b 0005000b
2016-05-23T10:10:44.028Z cpu1:33013)nvme:NvmeCore_GetStatus:462:Command failed: 0xa, INVALID FIELD IN CDB.
The failed command is a smart log page command, issued by some management tool of your host.
The LPA (Log Page Attribute) is 0, which means the SMART info is per drive, not per namespace.
However, the SMART command issued by the tool is set the namespace id as 1. This is rejected by device as INVALID FIELD IN CDB
I checked the kernel log, all the failed commands belong to this failure.
$ ~/temp/new$ grep nvmeCoreLogError . -r
./vmkernel.log:2016-05-23T10:10:44.028Z cpu1:33013)nvme:nvmeCoreLogError:333:command failed: 0x430f9bee7f00.
./vmkernel.log:2016-05-23T10:12:45.081Z cpu1:32781)nvme:nvmeCoreLogError:333:command failed: 0x430f9bee8400.
./vmkernel.log:2016-05-23T10:14:45.745Z cpu1:35399)nvme:nvmeCoreLogError:333:command failed: 0x430f9bee8900.
./vmkernel.log:2016-05-23T10:16:46.960Z cpu1:32775)nvme:nvmeCoreLogError:333:command failed: 0x430f9bee8e00.
./vmkernel.log:2016-05-23T10:18:48.128Z cpu1:33169)nvme:nvmeCoreLogError:333:command failed: 0x430f9bee9300.
./vmkernel.log:2016-05-23T10:20:49.052Z cpu1:34111)nvme:nvmeCoreLogError:333:command failed: 0x430f9bee9800.
but this error is reported by NVME driver?
Yes, because for admin command from management tool, NVMe driver just passes through it to device. It's the management tool's responsibility to make sure the admin command is constructed correctly.
but why Intel driver is silent? Error shows only VMWare driver, but anyway i forwarded question to HP team
Maybe the management tool only works for NVMe driver?