syoyo/tinyexr

The piz hufUncompress is comically bad...

Closed this issue · 1 comments

These two functions are completely bananas.

hufUnpackEncTable(&ptr, nCompressed - (ptr - compressed), im, iM, freq);
hufBuildDecTable(freq, im, iM, hdec);

hufUnpackEncTable first clears an array of 64K U64 values (512K). Doesn't need to, just does. It then reads up to 64K 6-bit values from the bit stream. It then stores these 6-bits into the U64 table. But for 99% of the file (there are usually only a couple hundred codes even a complicated piz) - it's just runs of zeros. So, clears the whole table out again.

Then it calls hufCanonicalCodeTable. This scans over the entire 512K again, building a histogram of the 6-bit values. Doesn't need to do that, since it could have just be done in hufUnpackEncTable, especially since 99% of the values are zero ANYWAY.

It then builds the lowest value from each one stores it back into the 64-bit value - now, the range is 16-bits+6-bits, so it still doesn't need a U64. It could also shrink the im->iM range here, so that we don't read a full 64K set of values when only a few hundred or so appear.

Then we call hufBuildDecTable() to build an acceleration table that is 14 bits - way too big for standard distributions, and too large to fit in most L1 caches. Then it spins over the entire 64K of symbols AGAIN (even though most don't appear), and then builds a little huffman chain for the long codes that mallocs and frees 1 -16 integers around 1200 times in the loop to build the table.

And on a 4K image, this is done around 135 times (for each block).

syoyo commented

huffman codes are grabbed from OpenEXR.

You can write your own one.
Contribution is alwyas welcome(Please send a PR)