fsck task blocked for more than x seconds
Opened this issue · 0 comments
Custom board with embedded linux. Linux kernel version 4.9.79, can't update to more recent.
Application CPU: Octeon3
Main memory device: Micron eMMC memory (8G)
One of the first commands executed while booting is fsck on top of one data partiton (r/w) one. Process will stuck once per 10 to 100 restart (power-cycles). Once fsck
process is stuck, device can't continue to boot as process is stuck in state D
Part of kernel log
[ 81.173679] INFO: task fsck.ext4:1165 blocked for more than 40 seconds.
[ 81.180303] Not tainted 4.9.79-ra2 #1
[ 81.184500] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 81.192329] fsck.ext4 D 0 1165 1 0x00100000
[ 81.197835] Stack : 800000002c2ea7b8 800000002c2ea7b8 0000000000000001 800000002c2ea7b8
[ 81.205903] 800000000114c900 0000000020000000 7fffffffffffffff 8000000001c70ba8
[ 81.213967] 0000000000000000 7fffffffffffffff 0000000000000002 0000000000000002
[ 81.222027] ffffffff80f79c18 ffffffff810b1330 0000000000000004 ffffffff80f7968c
[ 81.230091] 0000000000000000 ffffffff80f7c470 800000002b3dc738 800000002b6a2408
[ 81.238155] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.246220] 8000000001c70380 8000000001c70ba8 0000000000000000 7fffffffffffffff
[ 81.254285] 0000000000000002 0000000000000002 ffffffff80f79c18 ffffffff810b1330
[ 81.262344] 0000000000000004 ffffffff80f78bf8 0000000000000000 0000000000000002
[ 81.270409] ffffffff810b1328 8000000001fb7cb0 0000000000000000 ffffffff80f79c38
[ 81.278472] ...
[ 81.280940] Call Trace:[ 81.283216] [<ffffffff80f79c18>] 0xffffffff80f79c18
[ 81.288099] [<ffffffff80f7968c>] 0xffffffff80f7968c
[ 81.292976] [<ffffffff80f7c470>] 0xffffffff80f7c470
[ 81.297858] [<ffffffff80f79c18>] 0xffffffff80f79c18
[ 81.302736] [<ffffffff80f78bf8>] 0xffffffff80f78bf8
[ 81.307618] [<ffffffff80f79c38>] 0xffffffff80f79c38
[ 81.312494] [<ffffffff80f7a0bc>] 0xffffffff80f7a0bc
[ 81.317376] [<ffffffff809147f0>] 0xffffffff809147f0
[ 81.322253] [<ffffffff808d1498>] 0xffffffff808d1498
[ 81.327135] [<ffffffff80914904>] 0xffffffff80914904
[ 81.332013] [<ffffffff80915354>] 0xffffffff80915354
[ 81.336896] [<ffffffff809abd88>] 0xffffffff809abd88
[ 81.341772] [<ffffffff809a1460>] 0xffffffff809a1460
[ 81.346654] [<ffffffff8087f2e8>] 0xffffffff8087f2e8
[ 81.351533]
[ 81.353028] CPU: 0 PID: 13 Comm: khungtaskd Not tainted 4.9.79-ra2 #1
[ 81.359466] Stack : 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.367525] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.375583] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.383642] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.391702] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.399762] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.407822] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.415882] 0000000000000000 8000000001f87b70 0000000000000000 ffffffff80878b20
[ 81.423942] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 81.432001] 0000000000000000 ffffffff80878b20 0000000000000000 0000000000000000
[ 81.440059] ...
[ 81.442525] Call Trace:[ 81.444799] [<ffffffff80878b20>] 0xffffffff80878b20
[ 81.449678] [<ffffffff80878b20>] 0xffffffff80878b20
[ 81.454555] [<ffffffff80ab5524>] 0xffffffff80ab5524
[ 81.459432] [<ffffffff80875cbc>] 0xffffffff80875cbc
[ 81.464308] [<ffffffff8090bf80>] 0xffffffff8090bf80
[ 81.469186] [<ffffffff8090bc60>] 0xffffffff8090bc60
[ 81.474062] [<ffffffff808b1db8>] 0xffffffff808b1db8
[ 81.478940] [<ffffffff808b1cf0>] 0xffffffff808b1cf0
[ 81.483818] [<ffffffff808736d0>] 0xffffffff808736d0
[ 81.488696]
[ 81.490196] Kernel panic - not syncing: hung_task: blocked tasks
[ 81.496209] ---[ end Kernel panic - not syncing: hung_task: blocked tasks
The worst on this hung is the fact, kernel panic won't cause device reboot. The device is completely frozen up, until watchdog timeout is over and power cycle is executed again.
Translated stack trace attached.
trace.log
Any suggestions, what could be the root cause ?
Any suggestions, what to collect to solve this issue ?