Decoding triggers SIGKILL after a while of processing the input

[MarcSchoenefeld]

Running build from latest git pull, behavior consistent with binary from current Ubuntu package

$ ~/gits/astc-encoder/Source/astcenc-native   -dl sig1.astc /tmp/bla.png
Killed

Version is

$ ~/gits/astc-encoder/Source/astcenc-native   -v
astcenc v4.5.0, 64-bit sse2
Copyright (c) 2011-2023 Arm Limited. All rights reserved.

The reproducing bytes:

$ xxd sig1.astc
00000000: 13ab a15c 0a06 0199 0000 1800 0000 0040  ...\...........@
00000010: 4801 f202 bffc 2fff 41ff f87f fdff ffff  H...../.A.......
00000020: 08db 6e60 45e6 6fcb ffff ffef 1da2 3cd2  ..n`E.o.......<.
00000030: 4801 3243 2fff bff4 ffd0 ff07 ff7f ffff  H.2C/...........
00000040: fcfd ffff ffff ffff ffff ffff ffff ffff  ................
00000050: 7568 ca60 d43e f91f 775e 3f73 573f 735e  uh.`.>..w^?sW?s^
00000060: f1f9 3600 1821 1041 4cb8 3640 7fe4 fe70  ..6..!.AL.6@...p
00000070: 7600 26ff 81c3 8ff3 f913 3e9f c3f7 7bf8  v.&.......>...{.
00000080: fcfd ffff dfff dfff ffff ffff ffff ffff  ................
00000090: 7600 4aff 851f e1f3 297c 3f87 2fe7 f1c7  v.J.....)|?./...
000000a0: 0702 02fe 0100 004c 075c 2244 e980 2584  .......L.\"D..%.
000000b0: 4700 26ff 01c0 7049 0b29 25a6 a4c4 9492  G.&...pI.)%.....
000000c0: fcfd ffff ffff fffd ffbf ffff ffff ffff  ................
000000d0: 4801 7202 ffff f4ff c2ff 87ff 3ffc 7ff4  H.r.........?...
000000e0: 34d1 14a0 7cfc 2527 6072 717f 7ff8 7f70  4...|.%'`rq....p
000000f0: 08b1 4d60 4787 0fff f72f 3ff4 f2c7 bffe  ..M`G..../?.....
00000100: fcfd ffff ffff ffff ffff ffff efff ffff  ................

[MarcSchoenefeld]

OS on reproducing insance:

VERSION="22.04.2 LTS (Jammy Jellyfish)"

Linux ****** 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[MarcSchoenefeld]

There seems to be some endless loop around alloc_image:

Breakpoint 1, __GI___libc_malloc (bytes=616848) at ./malloc/malloc.c:3287
3287    in ./malloc/malloc.c
(gdb) mult
#0  __GI___libc_malloc (bytes=616848) at ./malloc/malloc.c:3287
#1  0x00007ffff7e1b98c in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055555555f2b5 in alloc_image(unsigned int, unsigned int, unsigned int, unsigned int) ()
#3  0x00005555555efa1a in astcenc_main(int, char**) ()
#4  0x00005555555f4731 in main ()

Breakpoint 1, __GI___libc_malloc (bytes=616848) at ./malloc/malloc.c:3287
3287    in ./malloc/malloc.c
(gdb) mult
#0  __GI___libc_malloc (bytes=616848) at ./malloc/malloc.c:3287
#1  0x00007ffff7e1b98c in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055555555f2b5 in alloc_image(unsigned int, unsigned int, unsigned int, unsigned int) ()
#3  0x00005555555efa1a in astcenc_main(int, char**) ()
#4  0x00005555555f4731 in main ()

Breakpoint 1, __GI___libc_malloc (bytes=616848) at ./malloc/malloc.c:3287
3287    in ./malloc/malloc.c
(gdb) mult
#0  __GI___libc_malloc (bytes=616848) at ./malloc/malloc.c:3287
#1  0x00007ffff7e1b98c in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055555555f2b5 in alloc_image(unsigned int, unsigned int, unsigned int, unsigned int) ()
#3  0x00005555555efa1a in astcenc_main(int, char**) ()
#4  0x00005555555f4731 in main ()

[solidpixel]

Thanks - I've reproduced the issue here.

The problem is caused by a corrupt file header (it's reporting an image dim_z of 4194304 pixels), so we're trying to allocate a lot of image slices as every slice is allocated separately. It eventually gets killed because it runs out of virtual memory. The bad failstate check is fixed in 3a2cda1 and will be in the 4.6.0 release.