How to generate valid json for fuzzer

[fernandezpablo85]

I have a simple function that does the following:

• given a []byte array, json-unmarshal it into a struct with two properties, say "a" and "b"
• if the value of "a" is < 5, for example, panic

I'm failing to get go-fuzz to trigger that (synthetic) panic, here's my Fuzz function:

package users

func Fuzz(data []byte) int {
	_, err := ReadStruct(data)
	if err != nil { 
		// this is likely a json.Unmarshal error, thus data is not valid json
		return 0
	}
	return 1
}

On corpus I've added a few examples like:

{"a": 10, "b": 200}

What am I missing? perhaps go-fuzz is not intended for finding these kind of issues?

[dgryski]

You're better off using something like https://github.com/google/gofuzz to fill in structures and Marshall those to JSON, then pass those randomized entities to your function.

[dvyukov]

Interesting. The mutation itself is not too complex, but I guess go-fuzz is lost testing all details of json parser...
I wonder if @thepudds https://github.com/thepudds/fzgo with rich signatures can help here. It should combine power of both coverage-guided fuzzing with property testing.

[thepudds]

@fernandezpablo85 are you using JSON in your example as an indirection, but your real goal is to fuzz your struct more directly?

If so, maybe fzgo could be useful to you. It is a layer on top of dvyukov/go-fuzz, where dvyukov/go-fuzz does the heavy lifting (so you get all the goodness of modern, coverage-guided fuzzing), and fzgo also happens to include preliminary support for fuzzing rich signatures as @dvyukov said.

Here is a quick example (which I think is similar to your example, if I followed?).

This finds a crasher within a few seconds.

Setup:

$ go get -u github.com/thepudds/fzgo
$ go get -u github.com/dvyukov/go-fuzz/...  # required if you don't already have this

Here is the fuzzing function, which sounds like it might be similar to the simple example you are trying (but directly using a struct rather than introducing a json-based indirection):

$ cat simplestruct.go
package simplestruct

type SimpleStruct struct {
	A int
	B int
}

func Fuzz_SimpleStruct(s SimpleStruct) {
	if s.A >= 1 && s.A < 5 {
		panic("bingo")
	}
}

Running:

$ fzgo test -fuzz=.
fzgo: detected rich signature for simplestruct.Fuzz_SimpleStruct
fzgo: building instrumented binary for simplestruct.Fuzz_SimpleStruct
fzgo: starting fuzzing simplestruct.Fuzz_SimpleStruct
fzgo: output in GOPATH\pkg\fuzz\corpus\github.com\thepudds\fzgo-corpus\simplestruct\Fuzz_SimpleStruct
2019/11/10 04:56:02 workers: 4, corpus: 8 (2s ago), crashers: 1, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s
2019/11/10 04:56:04 workers: 4, corpus: 8 (4s ago), crashers: 1, restarts: 1/0, execs: 0 (0/sec), cover: 307, uptime: 6s

That found a crasher right away.

Let's ask fzgo to run that crasher individually along with -v so that it prints the crashing arguments (which you can see below happens to be SimpleStruct{A:4, B:0}):

$ cp $GOPATH/pkg/fuzz/corpus/github.com/thepudds/fzgo-corpus/simplestruct/crashers/9b029004c325* $GOPATH/pkg/fuzz/corpus/github.com/thepudds/fzgo-corpus/simplestruct/corpus

$ fzgo test -fuzz=. -run=Corpus/9b029004c325 -v
=== RUN   TestCorpus/9b029004c3258b96365077bf5cc9990ff4268067
               arg 1:     simplestruct.SimpleStruct{A:4, B:0}
panic: bingo [recovered]

[thepudds]

@fernandezpablo85 On the other hand, maybe you have a pre-existing function that accepts a []byte of JSON data already, and you would prefer to use that pre-existing function (and perhaps can't conveniently take a struct as in my example above).

If so, you can use the rich signature support in fzgo to generate structures that your code then marshals to JSON. (This avoids spending a bunch of time creating and exploring invalid JSON; this alternative is somewhat closer to the suggestion from @dgryski above, but uses fzgo to retain the coverage-guided benefits of dvyukov/go-fuzz).

Here is a quick example of that alternative. This also finds a crasher within a few seconds. (The exact time though will of course depend on your exact examples; more complex examples will take longer).

Example:

package simplestruct

import "encoding/json"

type SimpleStruct struct {
	A int
	B int
}

// Fuzz a rich signature, which means this function is 
// handed a filled-in struct, but then marshal it to JSON for
// more convenient use in other pre-existing functions.
func Fuzz_SimpleStruct2(s SimpleStruct) {
	b, err := json.Marshal(s) // marshal to JSON as a convinience
	if err != nil {
		return
	}
	UseJSON(b)
}

// Let's pretend UseJSON is an existing function
// that takes a []byte of json data.
func UseJSON(b []byte) error {
	var s SimpleStruct
	err := json.Unmarshal(b, &s)
	if err != nil {
		return err
	}
	if s.A >= 1 && s.A < 5 {
		panic("bingo")
	}
	return nil
}

Sample run:

$ fzgo test -fuzz=.
fzgo: detected rich signature for simplestruct.Fuzz_SimpleStruct2
fzgo: building instrumented binary for simplestruct.Fuzz_SimpleStruct2
fzgo: starting fuzzing simplestruct.Fuzz_SimpleStruct2
fzgo: output in GOPATH\pkg\fuzz\corpus\github.com\thepudds\fzgo-corpus\simplestruct2\Fuzz_SimpleStruct2
2019/11/10 07:11:36 workers: 4, corpus: 12 (0s ago), crashers: 1, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s
2019/11/10 07:11:39 workers: 4, corpus: 12 (3s ago), crashers: 1, restarts: 1/0, execs: 0 (0/sec), cover: 849, uptime: 6s

The crasher found in this case:

$ cp $GOPATH/pkg/fuzz/corpus/github.com/thepudds/fzgo-corpus/simplestruct2/Fuzz_SimpleStruct2/crashers/18c4220359* $GOPATH/pkg/fuzz/corpus/github.com/thepudds/fzgo-corpus/simplestruct2/Fuzz_SimpleStruct2/corpus

$ fzgo test -fuzz=. -run=Corpus/18c42203590 -v
=== RUN   TestCorpus/18c42203590f36ef6f08a9585fa415d7b0240b02
               arg 1:     simplestruct.SimpleStruct{A:1, B:0}
panic: bingo [recovered]

[fernandezpablo85]

@thepudds (and @dvyukov) thanks so much for pointing me to fzgo and the examples! ❤️ this is exactly what I was looking for.

@dgryski gofuzz seems OK but I'd rather not loose the coverage-goodness of go-fuzz.

[fernandezpablo85]

@thepudds one last question, why doesn't fzgo use the int returns (to guide coverage) like go-fuzz?