/zig-std-lib-fuzzing

A set of fuzzers for fuzzing various parts of the Zig standard library

Primary LanguageZigBSD Zero Clause License0BSD

Fuzzing the Zig standard library

A set of fuzzers for fuzzing various parts of the Zig standard library. See 'Fuzzing Zig Code Using AFL++' for more information about the particular fuzzing setup used.

Current fuzzers:

  • tokenizer which calls std.zig.Tokenizer.next until it gets an eof token
  • parse which calls std.zig.Ast.parse and then std.zig.Ast.render
  • deflate which calls std.compress.flate.decompressor().reader().readAllAlloc()
  • deflate-puff which compares the results of puff.c to Zig's std.compress.flate.decompressor
  • deflate-roundtrip which sends the input through compressor, then through decompressor, and then checks that the output is the same as the input
  • json which calls std.json.parseFromSlice
  • sin which calls std.math.sin and compares the result to libc's sin/sinf
  • xz which calls std.compress.xz.decompress
  • xxhash which compares the results of xxhash.c to Zig's std.hash.xxhash implementation
  • zstandard which calls the std.compress.zstd decode, decodeAlloc, and decompressStream APIs.
  • zstandard-compare which compares the results of the zstd reference implementation to Zig's std.compress.zstd.decompress.decode implementation
  • zstandard-compare-alloc which compares the results of the zstd reference implementation to Zig's std.compress.zstd.decompress.decodeAlloc implementation
  • zstandard-compare-stream which compares the results of the zstd reference implementation to Zig's std.compress.zstd.decompressStream implementation
  • tar which uses std.tar.iterator to simulate an untar operation (but does not write to the filesystem)
  • tar-fs which calls std.tar.pipeToFileSystem (and actually writes to the filesystem)

Requires AFL++ with afl-clang-lto to be installed.

Building a fuzzer

Run zig build fuzz-<fuzzer name>, e.g. zig build fuzz-tokenizer

Running a fuzzer

The instrumented fuzzer will be installed to zig-out/bin/fuzz-<fuzzer name>. You'll probably also need to run mkdir outputs (if you're planning on using outputs as an output directory) before fuzzing. Here's a simple example of running the tokenizer fuzzer:

afl-fuzz -i inputs/tokenizer -o outputs/tokenizer -x dictionaries/zig.dict -- ./zig-out/bin/fuzz-tokenizer

(the -x option is not necessary but using a dictionary is recommended if possible)

See AFL++'s 'fuzzing the target' section for more recommendations to improve fuzzing effectiveness (using multiple cores, etc).

Debugging crashes

If a crash is found during fuzzing, the companion fuzz-<fuzzer name>-debug executable can be used to debug the crash. For example, for the tokenizer fuzzer, a stack trace could be gotten with:

$ ./zig-out/bin/fuzz-tokenizer-debug < 'outputs/tokenizer/default/crashes/id:000000,sig:06,src:000908+000906,time:117053,op:splice,rep:16'
thread 2730086 panic: index out of bounds
/home/ryan/Programming/zig/zig/build/lib/zig/std/zig/tokenizer.zig:408:34: 0x215131 in std.zig.tokenizer.Tokenizer.next (fuzz-tokenizer-debug)
            const c = self.buffer[self.index];
                                 ^
/home/ryan/Programming/zig/zig/build/lib/zig/std/zig/parse.zig:24:37: 0x20af60 in std.zig.parse.parse (fuzz-tokenizer-debug)
        const token = tokenizer.next();
                                    ^
...

Alternatively, the crash can be debugged via gdb:

gdb -ex 'set args < outputs/tokenizer/default/crashes/id:000000,sig:06,src:000908+000906,time:117053,op:splice,rep:16' ./zig-out/bin/fuzz-tokenizer-debug

Or valgrind:

valgrind ./zig-out/bin/fuzz-tokenizer-debug < 'outputs/tokenizer/default/crashes/id:000000,sig:06,src:000908+000906,time:117053,op:splice,rep:16'

zigescape can also be used to convert inputs into string literals for the creation of test cases (preferrably after using afl-tmin to minimize the input).

Bugs found / fixed

std.zig.Tokenizer

std.compress.flate

obsoleted Deflate implementations

std.compress.deflate (second version of Deflate)

std.compress.deflate (first version of Deflate)

std.math

std.compress.xz

std.compress.zstandard

std.tar

In upstream/third-party projects

facebook/zstd

Other tools/fuzzers

zstandard-verify

Requires the decodecorpus tool from zstd and the zstandard-verify tool from this repo (can be built with zig build tools). Run the following command to use it to continuously test the zstandard Zig decompressor with generated compressed .zst files:

./tools/zstandard-decodecorpus.sh /path/to/decodecorpus ./zig-out/bin/zstandard-verify