kernel oops under high load
Opened this issue · 6 comments
We have a number of machines using enhanceio, and we've been trying to nail down some lockups that happen under load. The load has increased substantially, and now we're seeing frequent lockups. I changed the kernel to panic on oops, so that these machines will reboot themselves, so it's not a crisis. :)
I was finally able to capture a lockup that happened early in a boot. Since this oops seems to render the machine locked up and useless, panicking seems like the right thing. Let me know if I can provide more info.
[ 115.652110] kernel BUG at /usr/src/enhanceio/EnhanceIO/Driver/enhanceio/eio_main.c:1518!
[ 115.660334] invalid opcode: 0000 [#1] SMP
[ 115.664866] Modules linked in: enhanceio_rand(O) enhanceio_lru(O) enhanceio_fifo(O) enhanceio(O) i2c_i801 coretemp
[ 115.675747] CPU: 0 PID: 1350 Comm: kworker/u48:3 Tainted: G W O 3.10.7-gentoo #3
[ 115.683923] Hardware name: Supermicro X8DTU/X8DTU, BIOS 2.1a 08/27/11
[ 115.691131] Workqueue: eio_callback eio_post_io_callback [enhanceio]
[ 115.697613] task: ffff88081f336190 ti: ffff88081e6ce000 task.ti: ffff88081e6ce000
[ 115.705174] RIP: 0010:[<ffffffffa002b6f2>] [<ffffffffa002b6f2>] eio_md_write+0x232/0x240 [enhanceio]
[ 115.714548] RSP: 0018:ffff88081e6cfdc8 EFLAGS: 00010286
[ 115.719922] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
[ 115.727120] RDX: 0000000000000020 RSI: 0000000000000046 RDI: 0000000000000282
[ 115.734323] RBP: ffffc90074ccc640 R08: ffffffff81820fd8 R09: 0000000000000000
[ 115.741526] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880750671ac0
[ 115.748726] R13: 0000000000001888 R14: 0000000000000213 R15: ffff880819588000
[ 115.755930] FS: 0000000000000000(0000) GS:ffff88083f200000(0000) knlGS:0000000000000000
[ 115.764104] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 115.769915] CR2: 00007fa90863a7b2 CR3: 0000000001b28000 CR4: 00000000000007f0
[ 115.777122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 115.784322] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 115.791526] Stack:
[ 115.793604] ffff880750a729c0 0000000100000282 ffff88083bde57c0 ffff88083bde57c0
[ 115.801347] ffff88083bc97c00 ffff880818ec0900 0000000000000000 0000000000000000
[ 115.809079] ffff8807505cdaa0 ffffffff810ae1d3 ffff880818ec0900 0000000000000000
[ 115.816818] Call Trace:
[ 115.819333] [<ffffffff810ae1d3>] ? process_one_work+0x153/0x420
[ 115.825405] [<ffffffff810af074>] ? worker_thread+0x114/0x370
[ 115.831219] [<ffffffff810aef60>] ? manage_workers.isra.26+0x2c0/0x2c0
[ 115.837819] [<ffffffff810b4743>] ? kthread+0xb3/0xc0
[ 115.842936] [<ffffffff810b4690>] ? __kthread_parkme+0x80/0x80
[ 115.848840] [<ffffffff81810d3c>] ? ret_from_fork+0x7c/0xb0
[ 115.854482] [<ffffffff810b4690>] ? __kthread_parkme+0x80/0x80
[ 115.860383] Code: 4c 89 7a 30 48 89 4a 38 48 89 55 40 48 c7 43 38 00 00 00 00 e9 16 ff ff ff 48 89 4d 38 49 89 6c 24 18 e9 3b fe ff ff 0f 0b 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 48 48 89 5c
[ 115.883270] RIP [<ffffffffa002b6f2>] eio_md_write+0x232/0x240 [enhanceio]
[ 115.890274] RSP <ffff88081e6cfdc8>
[ 115.894458] ---[ end trace 5bd50fb860d73366 ]---
[ 115.901224] Kernel panic - not syncing: Fatal exception
[ 116.895030] Rebooting in 300 seconds..
I'm going to add here that I haven't seen this happen on boxes that exclusively have drives 2TB and smaller. It has happened on machines with 3TB and 4TB drives. I don't know if it's another one of these 32 bit LBA bugs, but I thought it was noteworthy.
I've seen the same behaviour:
Feb 6 20:05:57 wirt kernel: [907547.614898] ------------[ cut here ]------------
Feb 6 20:05:57 wirt kernel: [907547.615168] kernel BUG at /var/lib/dkms/enhanceio/0.1/build/eio_main.c:1518!
Feb 6 20:05:57 wirt kernel: [907547.615580] invalid opcode: 0000 [#1] SMP
Feb 6 20:05:57 wirt kernel: [907547.615834] Modules linked in: btrfs(F) zlib_deflate(F) ufs(F) qnx4(F) hfsplus(F) hfs(F) minix(F) ntfs(F) msdos(F) jfs(F) xfs(F) libcrc32c(F) reiserfs(F) ext2(F) vesaf
b(F) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) gpio_ich x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmu
l ghash_clmulni_intel aesni_intel mxm_wmi aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd microcode joydev mei_me lpc_ich mei ioatdma wmi lp mac_hid parport raid10 raid456 async_raid6_recov asy
nc_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq raid1 raid0 igb isci multipath usbhid i2c_algo_bit ahci libsas usb_storage dca hid linear libahci ptp scsi_transport_sas pps_core
Feb 6 20:05:57 wirt kernel: [907547.653858] CPU: 0 PID: 20999 Comm: kworker/u49:0 Tainted: GF O 3.11.0-15-generic #23-Ubuntu
Feb 6 20:05:57 wirt kernel: [907547.676584] Hardware name: Thomas-Krenn.AG X9DR3-F/X9DR3-F, BIOS 3.0a 07/31/2013
Feb 6 20:05:57 wirt kernel: [907547.700132] Workqueue: eio_callback eio_post_io_callback [enhanceio]
Feb 6 20:05:57 wirt kernel: [907547.710692] task: ffff8808511faee0 ti: ffff880b448ac000 task.ti: ffff880b448ac000
Feb 6 20:05:57 wirt kernel: [907547.731170] RIP: 0010:[<ffffffffa03c54bc>] [<ffffffffa03c54bc>] eio_md_write+0x24c/0x250 [enhanceio]
Feb 6 20:05:57 wirt kernel: [907547.752336] RSP: 0018:ffff880b448add90 EFLAGS: 00010206
Feb 6 20:05:57 wirt kernel: [907547.762933] RAX: 0000000000000001 RBX: 000000000001facf RCX: 0000000000000001
Feb 6 20:05:57 wirt kernel: [907547.784307] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000282
Feb 6 20:05:57 wirt kernel: [907547.806288] RBP: ffff880b448addd0 R08: ffff88085fc14580 R09: 0000000000000001
Feb 6 20:05:57 wirt kernel: [907547.829061] R10: 0000000000000000 R11: 0000000000014580 R12: ffff88033cbb8a80
Feb 6 20:05:57 wirt kernel: [907547.852220] R13: 0000000000000001 R14: 0000000000000000 R15: ffffc9002eeaca38
Feb 6 20:05:57 wirt kernel: [907547.875239] FS: 0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000
Feb 6 20:05:57 wirt kernel: [907547.899255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 6 20:05:57 wirt kernel: [907547.910856] CR2: 00007f35579804e0 CR3: 0000000001c0e000 CR4: 00000000001427f0
Feb 6 20:05:57 wirt kernel: [907547.933950] Stack:
Feb 6 20:05:57 wirt kernel: [907547.944918] ffff88085190c000 0000000000000203 ffff88033cbb8700 ffff880cbbe16e70
Feb 6 20:05:57 wirt kernel: [907547.967166] ffff88085190c000 ffff880289f91960 0000000000000000 000000000001facf
Feb 6 20:05:57 wirt kernel: [907547.989756] ffff880b448ade20 ffffffffa03c5fc9 ffff880800000062 0000000001facf11
Feb 6 20:05:57 wirt kernel: [907548.012107] Call Trace:
Feb 6 20:05:57 wirt kernel: [907548.023179] [<ffffffffa03c5fc9>] eio_post_io_callback+0x739/0x950 [enhanceio]
Feb 6 20:05:57 wirt kernel: [907548.045135] [<ffffffff8107cfec>] process_one_work+0x17c/0x430
Feb 6 20:05:57 wirt kernel: [907548.056189] [<ffffffff8107dfac>] worker_thread+0x11c/0x3c0
Feb 6 20:05:57 wirt kernel: [907548.066831] [<ffffffff8107de90>] ? manage_workers.isra.25+0x2a0/0x2a0
Feb 6 20:05:57 wirt kernel: [907548.077500] [<ffffffff81084740>] kthread+0xc0/0xd0
Feb 6 20:05:57 wirt kernel: [907548.088052] [<ffffffff81084680>] ? kthread_create_on_node+0x120/0x120
Feb 6 20:05:57 wirt kernel: [907548.098605] [<ffffffff816f716c>] ret_from_fork+0x7c/0xb0
Feb 6 20:05:57 wirt kernel: [907548.108893] [<ffffffff81084680>] ? kthread_create_on_node+0x120/0x120
Feb 6 20:05:57 wirt kernel: [907548.119173] Code: 48 89 72 38 48 89 42 30 49 89 57 40 49 c7 46 38 00 00 00 00 e9 05 ff ff ff 49 89 4d 38 4d 89 6c 24 18 e9 2a fe ff ff 0f 0b 0f 0b <0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 ff 41 56 41
Feb 6 20:05:57 wirt kernel: [907548.151012] RIP [<ffffffffa03c54bc>] eio_md_write+0x24c/0x250 [enhanceio]
Feb 6 20:05:57 wirt kernel: [907548.161026] RSP <ffff880b448add90>
Feb 6 20:06:01 wirt kernel: [907548.183458] ---[ end trace 838f3d5ef81c5266 ]---
any comments on this by the devs? Is this project even alive?
I know it is bit too late to ask for details, but I was curious if you have details around following, I will try and reproduce the same on my local setup.
Could you please share following details...
- Cache configuration details (mode, block size of cache, policy), HDD, SSD sizes.
- Workload details, is it fio, vdbench or anything else for load generation?
- I presume it is writeback cache because the stack trace of panic show it is "eio_md_write" function stack which could only be run as part of write back cache.
- I assume the workload might be random and write intensive which lead to metadata writes in large numbers. I could not speculate what lead to CPU lockup but would be good to have a machine reproducing this issue.
Thank you.
I have seen the same issue on the ticket I just opened. I guess ill have to look at an alternative.
I have hit the same issue as noted here. I had set up Linux host as a iSCSI target on which there were about 10 Virtual machines running on VMware vsphere. Of the 10 VMs 5 were running vdbench, dt, and sqliosim. I have 3 SSDs in raid0 doing the duty of WB cache for a raid5 set of 5 HDDs.
~# eio_cli info
Cache Name : raidvol1_cache
Source Device : /dev/md126
SSD Device : /dev/md127
Policy : lru
Mode : Write Back
Block Size : 4096
Associativity : 256
State : normal
kernel: [2573885.309343] kernel BUG at /root/enhanceio/EnhanceIO/Driver/enhanceio/eio_main.c:1518!
kernel: [2573885.309445] invalid opcode: 0000 [#1] SMP
kernel: [2573885.309505] Modules linked in: ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_c
ore_iblock target_core_mod configfs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache enhanceio_rand(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap kvm_amd kvm crct10dif_
pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek nouveau serio_raw snd_hda_intel edac_core k10temp edac_mce_amd fam15h_power snd_hda_codec snd_hwdep m
xm_wmi video ttm joydev drm_kms_helper snd_pcm snd_page_alloc drm snd_timer sp5100_tco i2c_piix4 snd i2c_algo_bit soundcore 8021q garp stp mrp llc mac_hid wmi lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async
_xor async_tx hid_generic usbhid hid xor raid6_pq psmouse raid1 r8169 mii ixgbe raid0 dca mpt2sas ptp multipath ahci pps_core raid_class libahci linear scsi_transport_sas mdio
kernel: [2573885.310644] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF O 3.13.0-19-generic #40-Ubuntu
kernel: [2573885.310750] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2005 09/16/2013
kernel: [2573885.310865] Workqueue: eio_callback eio_post_io_callback [enhanceio]
kernel: [2573885.310931] task: ffff8807b6465fc0 ti: ffff8807ba7ce000 task.ti: ffff8807ba7ce000
kernel: [2573885.311031] RIP: 0010:[] [] eio_md_write+0x24c/0x250 [enhanceio]
kernel: [2573885.311140] RSP: 0018:ffff8807ba7cfd88 EFLAGS: 00010202
kernel: [2573885.311198] RAX: 0000000000000001 RBX: 00000000000088aa RCX: 00000000ffffffff
kernel: [2573885.311296] RDX: 0000000000000001 RSI: 000000002b1e2b1c RDI: 0000000000000286
kernel: [2573885.311394] RBP: ffff8807ba7cfdc8 R08: ffff88082ec94440 R09: 0000000000000001
kernel: [2573885.311493] R10: ffffffff8114f747 R11: ffffea001ee657c0 R12: ffff88065f53ae80
kernel: [2573885.311591] R13: 0000000000000001 R14: 0000000000000000 R15: ffffc900528b5fd0
kernel: [2573885.311690] FS: 00007f35f4fe9700(0000) GS:ffff88082ec80000(0000) knlGS:0000000000000000
kernel: [2573885.311792] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [2573885.311853] CR2: 00007fac050e4000 CR3: 00000007bd69b000 CR4: 00000000000407e0
kernel: [2573885.311953] Stack:
kernel: [2573885.312001] ffff8808019bb000 0000000000000213 ffff88065f53a280 ffff8807b995fe70
kernel: [2573885.312109] ffff8808019bb000 ffff8807ae03fc60 0000000000000000 000000000088aaad
kernel: [2573885.312217] ffff8807ba7cfe20 ffffffffa0521fb9 0000000000000000 ffff8808000088aa
kernel: [2573885.312325] Call Trace:
kernel: [2573885.312376] [] eio_post_io_callback+0x729/0x950 [enhanceio]
kernel: [2573885.312479] [] process_one_work+0x182/0x450
kernel: [2573885.312567] [] worker_thread+0x121/0x410
kernel: [2573885.312629] [] ? rescuer_thread+0x3e0/0x3e0
kernel: [2573885.312691] [] kthread+0xd2/0xf0
kernel: [2573885.312750] [] ? kthread_create_on_node+0x190/0x190
kernel: [2573885.312816] [] ret_from_fork+0x7c/0xb0
kernel: [2573885.312876] [] ? kthread_create_on_node+0x190/0x190
kernel: [2573885.312940] Code: 48 89 72 38 48 89 42 30 49 89 57 40 49 c7 46 38 00 00 00 00 e9 05 ff ff ff 49 89 4d 38 4d 89 6c 24 18 e9 2a fe ff ff 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 66 66 90 55 48 89 e5 41 57 4
kernel: [2573885.313286] RIP [] eio_md_write+0x24c/0x250 [enhanceio]
kernel: [2573885.313353] RSP
kernel: [2573885.314363] ---[ end trace 833c0a3658ba1937 ]---
kernel: [2573885.314589] BUG: unable to handle kernel paging request at ffffffffffffffd8
kernel: [2573885.314788] IP: [] kthread_data+0x10/0x20
kernel: [2573885.314935] PGD 1c11067 PUD 1c13067 PMD 0
kernel: [2573885.315192] Oops: 0000 [#2] SMP
kernel: [2573885.315372] Modules linked in: ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_c
ore_iblock target_core_mod configfs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache enhanceio_rand(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap kvm_amd kvm crct10dif_
pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek nouveau serio_raw snd_hda_intel edac_core k10temp edac_mce_amd fam15h_power snd_hda_codec snd_hwdep m
xm_wmi video ttm joydev drm_kms_helper snd_pcm snd_page_alloc drm snd_timer sp5100_tco i2c_piix4 snd i2c_algo_bit soundcore 8021q garp stp mrp llc mac_hid wmi lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async
_xor async_tx hid_generic usbhid hid xor raid6_pq psmouse raid1 r8169 mii ixgbe raid0 dca mpt2sas ptp multipath ahci pps_core raid_class libahci linear scsi_transport_sas mdio
kernel: [2573885.321176] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF D O 3.13.0-19-generic #40-Ubuntu
kernel: [2573885.321176] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF D O 3.13.0-19-generic #40-Ubuntu
kernel: [2573885.321323] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2005 09/16/2013
kernel: [2573885.321483] task: ffff8807b6465fc0 ti: ffff8807ba7ce000 task.ti: ffff8807ba7ce000
kernel: [2573885.321623] RIP: 0010:[] [] kthread_data+0x10/0x20
kernel: [2573885.321811] RSP: 0018:ffff8807ba7cfa78 EFLAGS: 00010002
kernel: [2573885.321910] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000002
kernel: [2573885.322049] RDX: 0000000000000001 RSI: 0000000000000002 RDI: ffff8807b6465fc0
kernel: [2573885.322188] RBP: ffff8807ba7cfa78 R08: 0000000000000000 R09: 0000000000000001
kernel: [2573885.322351] R10: ffffffff8133443a R11: ffffea001ee73780 R12: ffff88082ec94440
kernel: [2573885.322490] R13: 0000000000000002 R14: ffff8807b6465fb0 R15: ffff8807b6465fc0
kernel: [2573885.322630] FS: 00007f35f4fe9700(0000) GS:ffff88082ec80000(0000) knlGS:0000000000000000
kernel: [2573885.322772] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [2573885.322874] CR2: 0000000000000028 CR3: 00000007bd69b000 CR4: 00000000000407e0
kernel: [2573885.323013] Stack:
kernel: [2573885.323098] ffff8807ba7cfa90 ffffffff81083951 ffff8807b6465fc0 ffff8807ba7cfaf0
kernel: [2573885.323417] ffffffff81715249 ffff8807b6465fc0 ffff8807ba7cffd8 0000000000014440
kernel: [2573885.323765] 0000000000014440 ffff8807b6465fc0 ffff8807b64665f8 ffff8807b6465fb0
kernel: [2573885.324086] Call Trace:
kernel: [2573885.324177] [] wq_worker_sleeping+0x11/0x90
kernel: [2573885.324282] [] __schedule+0x589/0x7d0
kernel: [2573885.324384] [] schedule+0x29/0x70
kernel: [2573885.324485] [] do_exit+0x6df/0xa50
kernel: [2573885.324588] [] oops_end+0xa9/0x150
kernel: [2573885.324690] [] die+0x4b/0x70
kernel: [2573885.324823] [] do_trap+0x60/0x170
kernel: [2573885.324925] [] do_invalid_op+0xa2/0x100
kernel: [2573885.325030] [] ? eio_md_write+0x24c/0x250 [enhanceio]
kernel: [2573885.325139] [] ? ttwu_do_activate.constprop.74+0x5d/0x70
kernel: [2573885.325247] [] ? try_to_wake_up+0x1fa/0x2c0
kernel: [2573885.325352] [] ? kfree+0xfd/0x130
kernel: [2573885.325452] [] invalid_op+0x1e/0x30
kernel: [2573885.325554] [] ? mempool_free_slab+0x17/0x20
kernel: [2573885.325661] [] ? eio_md_write+0x24c/0x250 [enhanceio]
kernel: [2573885.325771] [] ? eio_md_write+0x1ab/0x250 [enhanceio]
kernel: [2573885.325882] [] eio_post_io_callback+0x729/0x950 [enhanceio]
kernel: [2573885.326024] [] process_one_work+0x182/0x450
kernel: [2573885.326129] [] worker_thread+0x121/0x410
kernel: [2573885.326232] [] ? rescuer_thread+0x3e0/0x3e0
kernel: [2573885.326341] [] kthread+0xd2/0xf0
kernel: [2573885.326446] [] ? kthread_create_on_node+0x190/0x190
kernel: [2573885.326553] [] ret_from_fork+0x7c/0xb0
kernel: [2573885.326654] [] ? kthread_create_on_node+0x190/0x190
kernel: [2573885.326760] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 87 a8 03 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90
kernel: [2573885.329987] RIP [] kthread_data+0x10/0x20
kernel: [2573885.330134] RSP
kernel: [2573885.330227] CR2: ffffffffffffffd8
kernel: [2573885.330319] ---[ end trace 833c0a3658ba1938 ]---
kernel: [2573885.330416] Fixing recursive fault but reboot is needed!
kernel: [2573905.625082] ------------[ cut here ]------------
kernel: [2573905.625186] WARNING: CPU: 2 PID: 976 at /build/buildd/linux-3.13.0/kernel/watchdog.c:245 watchdog_overflow_callback+0x9c/0xd0()
kernel: [2573905.626983] Watchdog detected hard LOCKUP on cpu 2
kernel: [2573905.627048] Modules linked in: ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod configfs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache enhanceio_rand(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek nouveau serio_raw snd_hda_intel edac_core k10temp edac_mce_amd fam15h_power snd_hda_codec snd_hwdep mxm_wmi video ttm joydev drm_kms_helper snd_pcm snd_page_alloc drm snd_timer sp5100_tco i2c_piix4 snd i2c_algo_bit soundcore 8021q garp stp mrp llc mac_hid wmi lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic usbhid hid xor raid6_pq psmouse raid1 r8169 mii ixgbe raid0 dca mpt2sas ptp multipath ahci pps_core raid_class libahci linear scsi_transport_sas mdio
kernel: [2573905.632902] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF D O 3.13.0-19-generic #40-Ubuntu
kernel: [2573905.633049] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2005 09/16/2013
kernel: [2573905.633209] 0000000000000009 ffff88082ec86c38 ffffffff81711075 ffff88082ec86c80
kernel: [2573905.633526] ffff88082ec86c70 ffffffff810662cd ffff880802e68000 0000000000000000
kernel: [2573905.633842] ffff88082ec86d88 0000000000000000 ffff88082ec86ef8 ffff88082ec86cd0
kernel: [2573905.634185] Call Trace:
kernel: [2573905.634272] [] dump_stack+0x45/0x56
kernel: [2573905.634467] [] warn_slowpath_common+0x7d/0xa0
kernel: [2573905.634571] [] warn_slowpath_fmt+0x4c/0x50
kernel: [2573905.634674] [] ? restart_watchdog_hrtimer+0x50/0x50
kernel: [2573905.634780] [] watchdog_overflow_callback+0x9c/0xd0
kernel: [2573905.634886] [] __perf_event_overflow+0x8e/0x240
kernel: [2573905.634990] [] perf_event_overflow+0x14/0x20