Support for skippable frames
KillingSpark opened this issue · 4 comments
KillingSpark commented
This implementation does not currently support skippable frames which are required for a spec compliant implementation
https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#skippable-frames
This relates to the seekable format
KillingSpark commented
phord commented
I found out that pzstd (no longer being developed) produces skippable frames.
git/opc/zstd-rs(skipable_frames)» pzstd LICENSE
LICENSE : 64.74% ( 1075 => 696 bytes, LICENSE.zst)
git/opc/zstd-rs(skipable_frames)» cmp LICENSE <(zstd -dc LICENSE.zst)
git/opc/zstd-rs(skipable_frames)» md5sum LICENSE
13df02d6656dafde4ac24f5a0b25f86d LICENSE
git/opc/zstd-rs(skipable_frames)» zstdcat LICENSE.zst | md5sum
13df02d6656dafde4ac24f5a0b25f86d -
git/opc/zstd-rs(skipable_frames)» target/release/zstd -d -c LICENSE.zst | md5sum
File: LICENSE.zst
Found a skippable frame with magic number: 407710288 and size: 4
100 % done
Decoded frames: 1 bytes: 1075
1 of 1 checksums are ok!
13df02d6656dafde4ac24f5a0b25f86d -
TEST PASSED 👍
phord commented
Here's the error the original code produces when skippable frames are encountered:
git/opc/zstd-rs(master)» pzstd foo
foo : 10.26% (2043388670 => 209694227 bytes, foo.zst)
git/opc/zstd-rs(master)» ls -lah foo*
-rw-rw-r-- 1 phord phord 2.0G Mar 7 12:51 foo
-rw-rw-r-- 1 phord phord 200M Mar 15 11:03 foo.zst
git/opc/zstd-rs(master)» target/release/zstd -d -c foo.zst |wc -l
File: foo.zst
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: FrameCheckError(WrongMagicNum { got: 407710288 })', src/bin/zstd.rs:60:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Also, it turns out that pzstd actually inserts lots of skippable frames on large files.
git/opc/zstd-rs(skipable_frames)» target/release/zstd -d -c foo.zst > /dev/null
File: foo.zst
Found a skippable frame with magic number: 407710288 and size: 4
0 % doneFound a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
1 % doneFound a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
2 % doneFound a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
3 % doneFound a skippable frame with magic number: 407710288 and size: 4
:
98 % doneFound a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
99 % doneFound a skippable frame with magic number: 407710288 and size: 4
Found a skippable frame with magic number: 407710288 and size: 4
100 % done
Decoded frames: 244 bytes: 2043388670
244 of 244 checksums are ok!
13239317
git/opc/zstd-rs(skipable_frames)» target/release/zstd -d -c foo.zst 2>&1 | grep -c "skippable frame with magic number"
244
KillingSpark commented
Nice! Thanks a lot for double checking, much appreciated.