/erlfuzz

erlfuzz is a fuzzer for the Erlang ecosystem

Primary LanguageRustApache License 2.0Apache-2.0

Erlfuzz

erlfuzz is a small standalone generator of random erlang programs used to fuzz the erlang compiler and VM (and other tools such as erlfmt, dialyzer, eqWAlizer, etc.). It does not currently use coverage information or do anything especially clever, except for carefully handling Erlang's scoping rules.

Requirements

Dependencies of the fuzzer itself:

Dependencies of the scripts wrapping fuzzing targets:

  • Bash
  • GNU Parallel
  • Timeout (part of GNU coreutils)

The fuzzer has been tested on both Linux and MacOS. On the latter, the scripts need a small tweak to run: removing or replacing calls to the ulimit command.

How to build

On a machine with access to the internet: cargo build --release

How to use

With a provided script

If the goal is to fuzz erlc & erl, the easiest way is to use the provided run_erl_debug.sh script from erlfuzz/ as follows:

mkdir -p out
mkdir -p interesting
seq 1000000 | parallel --line-buffer "./target/release/erlfuzz fuzz -c ./scripts/run_erl_debug.sh --tmp-directory out --interesting-directory interesting test{}"

This will repeatedly run the fuzzer, store its output in a file in out/, call erlc and erl on that file, and then either delete it or move it to interesting/ if it triggered a crash. It will automatically make use of all cores on your machine. If that is not what you want, you can limit it to 5 cores (for example) by passing -j 5 to parallel. It will automatically stop after 1M iterations, just increase the parameter to seq if you want more.

I recommend running this in a window controlled by tmux, screen, etc., so it can be detached and survive the end of whatever terminal/ssh session you are using.

There are similar scripts for fuzzing other tools such as erlfmt, dialyzer, or eqwalizer. For some of these, it may be required to pass additional options to erlfuzz:

  • run_eqwalizer_once.sh:
    • --disable-shadowing
  • run_gradualizer_once.sh:
    • --disable-map-comprehensions
    • --disable-maybe
  • run_infer_once.sh:
    • --disable-map-comprehensions
    • --disable-maybe
    • --disable-shadowing
    • --wrapper for-infer
  • verify_erl_jit.sh:
    • --deterministic
    • --wrapper printing
  • verify_erlc_opts.sh:
    • --deterministic
    • --wrapper printing
  • verify_infer_against_erl.sh:
    • --deterministic
    • --disable-map-comprehensions
    • --disable-maybe
    • --disable-shadowing
    • --wrapper printing

With another script

erlang fuzz can be used to fuzz any other command that accepts an Erlang file. A return code of 0 must be used to indicate that the program ran successfully, and non-0 to report that a bug was found. I strongly recommend wrapping the program being fuzzed by a script that uses timeout and ulimit -m; you can look at run_erl_debug.sh for inspiration. Also consider using stronger sandboxing (e.g. with cgroups) if the program under test is likely to have dangerous side effects.

Manually

Instead of using erlfuzz fuzz on a script like above, you can use erlfuzz to generate an Erlang file on stdout, which you can then use however you want. See erlfuzz generate for details.

Testcase minimization

Testcases found by erlfuzz can be automatically reduced, by using erlfuzz reduce, and passing it the seed used to generate the testcase. If any additional option was passed during generation time, it must also be passed at reduction time. Both the seed and all relevant options are listed in a comment at the top of the testcase.

For example:

./target/release/erlfuzz reduce -c ./scripts/run_erl_debug.sh --tmp-directory out --minimized-directory minimized --seed 3137934125722527840 foobar

Would reduce the testcase that was generated with seed 3137934125722527840, use out/foobar.erl as a scratch file, and store the result in minimized/foobar.erl.

Depending on the formatting of the error messages that are output by the tool under test, it can be necessary to adjust the similarity heuristic with the --max-distance or the --num-lines options. Dialyzer for example includes non-deterministic numbers (timestamp, process id) in its error messages, so --max-distance 10 is required for erlfuzz to ignore those.

You can ask erlfuzz to automatically reduce the bugs it finds as they are found, by using erlfuzz fuzz-and-reduce. This is a tiny wrapper around doing erlfuzz fuzz followed by erlfuzz reduce, and needs all of the same arguments.

Debugging erlfuzz

If erlfuzz itself is misbehaving, one way to investigate is to set the environment variable RUST_LOG=debug or RUST_LOG=trace (even more verbose) to make it emit a log of its actions to stderr. It is possible to set different modules to different levels of logging, for example RUST_LOG="warn,erlfuzz::reducer=info" will set all modules to level "warn" except for the testcase reducer which it will set to level info. See crate env_logger for details. The levels from least to most verbose are:

  • error (default)
  • warn
  • info
  • debug
  • trace

I recommend the following setting to get a log of progress during reduction: RUST_LOG="debug,erlfuzz::environment=warn".

License

erlfuzz is Apache licensed.

Current limitations

None of the following is currently generated by the fuzzer:

  • receive constructs (and more generally the code generated is currently non-concurrent)
  • preprocessor commands
  • map iterators

Erlfuzz also only knows about some bifs.

Some of the bugs found so far

BEAM VM:

erlc:

dialyzer:

erlfmt:

eqWAlizer:

inferl:

Gradualizer