A Solidity test generator based on the Branching Tree Technique.
Warning
bulloak
is still 0.*.*
, so breaking changes may occur at any time. If you must depend on bulloak
, we recommend pinning to a specific version, i.e., =0.y.z
.
cargo install bulloak
The following VSCode extensions are not essential but they are recommended for a better user experience:
- Tree: syntax highlighting for
.tree
files - Ascii Tree Generator: convenient way to generate ASCII trees
bulloak
implements two commands:
bulloak scaffold
bulloak check
Say you have a foo.tree
file with the following contents:
FooTest
└── When stuff is called // Comments are supported.
└── When a condition is met
└── It should revert.
└── Because we shouldn't allow it.
You can use bulloak scaffold
to generate a Solidity contract
containing modifiers and tests that match the spec described in
foo.tree
. The following will be printed to stdout
:
$ bulloak scaffold foo.tree
pragma solidity 0.8.0;
contract FooTest {
modifier whenStuffIsCalled() {
_;
}
function test_WhenAConditionIsMet()
external
whenStuffIsCalled
{
// It should revert.
// Because we shouldn't allow it.
}
}
You can use the -w
option to write the generated contracts
to the file system. Say we have a bunch of .tree
files in
the current working directory. If we run the following:
$ bulloak scaffold -w ./**/*.tree
bulloak
will create a .t.sol
file per .tree
file and write
the generated contents to it.
If a .t.sol
file's title matches a .tree
in the same directory,
then bulloak
will skip writing to that file. However, you may
override this behaviour with the -f
flag. This will force bulloak
to overwrite the contents of the file.
$ bulloak scaffold -wf ./**/*.tree
You can use bulloak check
to make sure that your Solidity files
match your spec. For example, any missing tests will be reported
to you.
Say you have the following spec:
HashPairTest
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.
And a matching Solidity file:
pragma solidity 0.8.0;
contract HashPairTest {
function test_ShouldNeverRevert() external {
// It should never revert.
}
function test_WhenFirstArgIsSmallerThanSecondArg() external {
// It should match the result of `keccak256(abi.encodePacked(a,b))`.
}
}
This Solidity file is missing the tests for When first arg is bigger than second arg
,
which would be reported after running bulloak check
, like so:
•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
check failed: couldn't find a corresponding element for "test_WhenFirstArgIsBiggerThanSecondArg" in the Solidity file
file: /Users/alexfertel/stuff/test.tree | line: 5
The following rules are currently implemented:
- A Solidity file matching the spec file must exist and be readable.
- The spec and the Solidity file match if the difference between their names is only
.tree
&.t.sol
.
- The spec and the Solidity file match if the difference between their names is only
- There is a contract in the Solidity file and its name matches the root node of the spec.
- Every construct, as it would be generated by
bulloak scaffold
, is present in the Solidity file. - The order of every construct, as it would be generated by
bulloak scaffold
, matches the spec order.- Any valid Solidity construct is allowed and only construct that would be generated by
bulloak scaffold
are checked. This means that any number of extra functions, modifiers, etc. can be added to the file.
- Any valid Solidity construct is allowed and only construct that would be generated by
Another feature of bulloak
is reporting errors in your input trees.
For example, say you have a buggy foo.tree
file, which is missing a
└
character. Running bulloak scaffold foo.tree
would report the error like this:
•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
bulloak error: unexpected `when` keyword
── when the id references a null stream
^^^^
--- (line 2, column 4) ---
file: foo.tree
bulloak scaffold
scaffolds Solidity test files based on .tree
specifications
that follow the Branching Tree Technique.
Currently, there is on-going discussion on how to handle different edge-cases to better empower the Solidity community. This section is a description of the current implementation of the compiler.
- Condition:
when/given
branches of a tree. - Action:
it
branches of a tree. - Action Description: Children of an action.
Each tree
file should describe a function under test. Trees follow these rules:
- The line at the top of the file is the name of the contract.
bulloak
expects you to use├
and└
characters to denote branches.- If a branch starts with either
when
orgiven
, it is a condition.when
andgiven
are interchangeable.
- If a branch starts with
it
, it is an action.- Any child branch an action has is called an action description.
- Keywords are case-insensitive:
it
is the same asIt
andIT
. - Anything starting with a
//
is a comment and will be stripped from the output.
Take the following Solidity function:
function hashPair(bytes32 a, bytes32 b) private pure returns (bytes32) {
return a < b ? hash(a, b) : hash(b, a);
}
A reasonable spec for the above function would be:
HashPairTest
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.
There is a top-level action which will generate a test to check the function invariant that it should never revert.
Then, we have the two possible preconditions: a < b
and a >= b
. Both branches end in an action that will make bulloak scaffold
generate the respective test.
Note the following things:
- Actions are written with ending dots but conditions are not. This is because actions support any character, but conditions don't. Since conditions are transformed into modifiers, they have to be valid Solidity identifiers.
- You can have top-level actions without conditions. Currently,
bulloak
also supports actions with sibling conditions, but this might get removed in a future version per this discussion. - The root of the tree will be emitted as the name of the test contract.
There are a few things to keep in mind about the scaffolded Solidity test:
- The contract filename is the same as the
.tree
but with a.t.sol
extension. E.g.test.tree
would correspond totest.t.sol
. - Test are emitted in the order their corresponding actions appear in the
.tree
file. - We generate one modifier per condition, except for leaf condition nodes.
- Test names follow Foundry's best practices.
Please refer to CONTRIBUTING.md.
These are the current steps taken to publish bulloak
:
- Bump the version field in Cargo.toml.
- Create the corresponding git tag named after the version.
- Update the CHANGELOG.md file with
git cliff -o CHANGELOG.md
. - Run
cargo publish --dry-run
to make sure that everything looks good. - Commit & push the changes.
- Run
cargo publish
.
This project is licensed under either of:
- Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0).
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT).