vmarchese/gremlins

A mutation testing tool for Go.

Gremlins

WARNING: Gremlins isn't currently supported on Windows.

Gremlins is a mutation testing tool for Go. It has been made to work well on smallish Go modules, for example microservices, on which it helps validate the tests, aids the TDD process and can be used as a CI quality gate. As of now, Gremlins doesn't work very well on very big Go modules, mainly because a run can take hours to complete.

• What is Mutation Testing
• How to Use Gremlins
  • Obtain and install
  • Usage
• Supported Mutations
  • Conditionals Boundaries
  • Conditionals Negation
  • Increment Decrement
  • Invert Negatives
  • Arithmetic Base
• Current Limitations
• What Inspired Gremlins
• Other Mutation Testing Tools for Go
• Contributing
• License

What is Mutation Testing

Code coverage is unreliable as a measure of test quality. It is too easy to have tests that exercise a piece of code but don't test anything at all. Mutation testing works by mutating the code exercised by the tests and verifying if the mutation is caught by the test suite. Imagine gremlins going into your code and messing around: will your test suit catch their damage?

Here is a nice intro to mutation testing.

How to use Gremlins

Obtain and install

Linux

Download a .deb or .rpm file from the release page and install with dpkg -i and rpm -i respectively.

MacOS

On macOS, you can use Homebrew to install by first tapping the repository. As of now, we use a homebrew tap.

brew tap go-gremlins/tap
brew install gremlins

Manual

• Download the binary appropriate for your platform from the release page .
• Put the gremlins executable somewhere in your PATH (ex. /usr/local/bin).

From source

To build Gremlins you need the Go compiler, make and golangci-lint for linting. You can clone the Gremlins repository and then build it:

git clone https://github.com/go-gremlins/gremlins.git

Ad then:

cd gremlins
make

Usage

To execute a mutation test run, from the root of a Go module execute:

$ gremlins unleash

Gremlins only tests mutations of parts of the code already covered by test cases. If a mutant is not covered, why bother testing? You already know it will not be caught. In any case, Gremlins will report which mutations aren't covered.

If the Go test run needs build tags, they can be passed along:

$ gremlins unleash --tags "tag1,tag2"

To perform the analysis without actually running the tests:

$ gremlins unleash --dry-run

Gremlins will report each mutation as:

• RUNNABLE: In dry-run mode, a mutation that can be tested.
• NOT COVERED: A mutation not covered by tests; it will not be tested.
• KILLED: The mutation has been caught by the test suite.
• LIVED: The mutation hasn't been caught by the test suite.
• TIMED OUT: The tests timed out while testing the mutation: the mutation actually made the tests fail, but not explicitly.
• NOT VIABLE: The mutation makes the build fail.

Supported mutations

Conditionals Boundaries

Original	Mutated
>	>=
>=	>
<	<=
<=	<

Example:

if a > b {
// Do something
}

will be changed to

if a < b {
// Do something
}

Conditionals Negation

Original	Mutated
==	!=
!=	==
>	<=
<=	>
<	>=
>=	<

Example:

if a == b {
// Do something
}

will be changed to

if a != b {
// Do something
}

Increment Decrement

Original	Mutated
++	--
--	++

Example:

func incr(i int) int
return i++
}

will be changed to

func incr(i int) int {
return i--
}

Invert Negatives

It will invert negative numbers.

Example:

func negate(i int) int {
return -i
}

will be changed to

func negate(i int) int {
return +i
}

Arithmetic Base

Original	Mutated
+	-
-	+
*	/
/	*
%	*

Example:

a := 1 + 2

will be changed to

a := 1 - 2

Current limitations

There are some limitations on how Gremlins works right now, but rest assured we'll try to make things better.

• Gremlins can be run only from the root of a Go module and will run all the test suite. This is a problem if the tests are especially slow.
• For each mutation, Gremlins will run all the test suite. It would be better to only run the test cases that actually cover the mutation.
• Gremlins doesn't support custom test commands; if you have to do anything different from go test [-tags t1 t2] ./... to run your test suite, most probably it will not work with Gremlins.
• There is no way to implement custom mutations.
• It is not tested on Windows as of now and most probably it will not work there.

What inspired Gremlins

Mutation testing exists since the early days of computer science, so loads of papers and articles do exists. Gremlins is inspired from those.

Among the existing mutation testing tools, Gremlins is inspired especially by PITest.

Other Mutation Testing tools for Go

There is not much around, except from:

• go-mutesting

Contributing

See contributing.

License

Gremlins is released under the Apache 2.0 License.