Kernel panic on huge files?

Question

Kernel panic on huge files?

Closed this issue 5 years ago · 10 comments

Just for reference, if it happens to other people:

I ran afsctool on directories having some few GB files using the following command (if I understand, it works in single-threaded mode):
afsctool -cvvv -9 -L …/folders/at/wherever

When processing the huge files, memory compression and swapfile usage went up quite extremely, but of course it's expected. The unexpected thing is the kernel panic. It happened twice so far. I have no clue if the error happened in 's memory compression or the file compression part.

Panic happens here (just for fun):
https://opensource.apple.com/source/xnu/xnu-4570.71.2/osfmk/vm/vm_fault.c.auto.html

if ((map == kernel_map) && (caller_prot & VM_PROT_WRITE)) {
	if (compressor_map) {
		if ((vaddr >= vm_map_min(compressor_map)) && (vaddr < vm_map_max(compressor_map))) {
			panic("Write fault on compressor map, va: %p type: %u bounds: %p->%p", (void *) vaddr, caller_prot, (void *) vm_map_min(compressor_map), (void *) vm_map_max(compressor_map));

Shortened panic report:

*** Panic Report ***
panic(cpu 0 caller 0xffffff80274ed17b): "Write fault on compressor map, va: 0xffffff9128ce8000 type: 3 bounds: 0xffffff821247d000->0xffffff92126bd000"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.71.45/osfmk/vm/vm_fault.c:3523
Backtrace (CPU 0), Frame : Return Address
0xffffff803aa03170 : 0xffffff802746bf96
…
0xffffff803aa03fa0 : 0xffffff802741dec2
Kernel Extensions in backtrace:
com.apple.filesystems.hfs.kext(407.50.6)[…]@0xffffff7fa9b3a000->0xffffff7fa9ba0fff
dependency: com.apple.filesystems.hfs.encodings.kext(1)[…]@0xffffff7fa9b32000
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib(1.0)[…]@0xffffff7faa77c000->0xffffff7faa788fff

BSD process name corresponding to current thread: afsctool

Mac OS version:
17G7024 /// == High Sierra 10.13.6
Kernel version:
Darwin Kernel Version 17.7.0: Wed Apr 24 21:17:24 PDT 2019; root:xnu-4570.71.45~1/RELEASE_X86_64

Model: Macmini6,2, BootROM 278.0.0.0.0, 4 processors, Intel Core i7, 2.3 GHz, 16 GB, SMC 2.8f1

Answer 1 · 2019-07-23T19:38:42.000Z

this seems to be virtual memory. is your hard drive definitely ok? could it be failing?

how much VM was being used?

Answer 2 · 2019-07-23T19:47:24.000Z

how much VM was being used?

And (more) importantly, how much free space do you have on the drive? If you're short on space it can be a good idea to deactivate the verify/backup feature (-n option); this will probably reduce memory requirements too.

Answer 3 · 2019-07-23T20:14:46.000Z

I don't remember where did I experiment first time, but this time GSmartControl warned about soft read errors, so I'll have to experiment more with another disk. Both the external drive hosting the large file and the startup disk have plenty of space (30+GB).

Anyway, I ran an experiment with just one file (a copy of an OSX installer image around 6-7GB). It took quite long, it ate all RAM, etc., but it apparently compressed the file, released all the memory. But at the verification phase (I think) it kernelpanicked again. I compared the compressed copy with the original with another tool, and … panic again. Copied the file to another drive, tried to open it, zip it, but they complained about "invalid argument" or such. afsctool -d image.dmg said that the (otherwise almost incompressible) file occupies 4GB on disk. Finder's Get Info said 6GB (compressed) 2GB on disk.

So it's funny. But as I said I have to test this more. Anyway, thanks for replying!

Answer 4 · 2019-07-23T21:48:13.000Z

I thought I disabled the support for files > 2Gb?! Such files could be stored in a compressed disk image, which also means the much more capable bzip2 compressor can be used.

Answer 5 · 2019-07-23T22:10:13.000Z

I actually made a service to Finder to (re)compress disk images, but this time I just sent afsctool to every old & forgotten file in a folder, because why not. This sh*t is somehow addictive :)

Answer 6 · 2019-08-14T14:58:47.000Z

Can you please check with the latest commit?

Answer 7 · 2019-08-15T15:52:59.000Z

Can you please check with the latest commit?

So far it works and when file is too big, it properly emits with unsupportable size.
Thanks! :)

BTW, maybe this useful tool could be promoted among developers who offer .dmg images with "drag this to the Applications folder to install". Gimp, LibreOffice, VirtualBox are compressed to less than the half of their original sizes. My top (by size) 3rd party apps are 29GB uncompressed, but became 14GB after compression. On small  SSDs, this can be quite a huge saving.

Answer 8 · 2019-08-16T10:36:43.000Z

BTW, maybe this useful tool could be promoted among developers who offer .dmg images with "drag this to the Applications folder to install".

I agree, perhaps @biziclop can ask the most popular DMG creation apps to add support?

Answer 9 · 2019-08-16T12:40:54.000Z

I'm inclined to leave that to users of said tools, which I'm not. After all, why would the devs add support for something for which there's no user demand?

Answer 10 · 2019-08-16T13:08:01.000Z

Yes, I meant @biziclop who raised the point.

I also think that Sparkle Updater framework could be educated in the same way about HFS+ compression.