Add coverage guided fuzzing
Closed this issue · 10 comments
It would be really nice if we could have coverage guided fuzzing.
- Do a bunch of fuzzing.
- For each input, record the code coverage from that input.
- If the input exercises new code, add it to the corpus.
We'll still want random fuzzing, but this will help us make sure we at least cover as many error paths as possible.
Originally posted by @Stebalien in #219 (comment)
We should first add test coverage to CI! @masih would you look into this?
I predict this will show some holes in our unit testing at the level of explicitly testing message sequences given to the participant. This is a gap between the simple data structure unit tests and sim-level exercises we currently have.
Ah, actually, go fuzzing already works this way. It just stores the entries in $(go env GOCACHE)/fuzz. We need to upload this somewhere when the test finishes (success or failure) and download it afterwards.
We could use caching but... ideally we'd push it to a git repo somewhere. @Kubuxu should be able to help here.
We should first add test coverage to CI! @masih would you look into this?
Will do; that should be quick.
ideally we'd push it to a git repo somewhere
I would just push those to testdata next to the tests. That would be easier; no?
It might be more optimal to move away from seed fuzzing and instead fuzze an array of latencies or something similar, but seed-based fuzzing is a great initial step.
Giving more precise control over latencies to the fuzzer would make it more expressive, allowing it to test more scenarios and allow it to make smaller changes to the scenario as it evolves.
We could use caching but... ideally we'd push it to a git repo somewhere. @Kubuxu should be able to help here.
In my opinion, we should store minimized fuzz corpus in this repo. The separate repo works but is a bit hacky, and I've done it in the past mostly because the fuzzer used was AFL, and I was fuzzing complex scenarios with large churn.
Hmm, looks like go doesn't support corpus minimization right now :/
-fuzzminimizetime not it?
Nope, that is for minimizing the crash reproducer (essentially, it will take the crasher and fuzz it to reduce its size while keeping the crash).