jmgarnier/test-boosters

Auto Parallelization - runs test files in multiple jobs

Test Boosters

Auto Parallelization — runs test files in multiple jobs

• Installation

Test Booster basics:

• What are Test Boosters
• Split Configuration
• Leftover Files

Test Boosters:

• RSpec Booster
• Cucumber Booster
• Minitest Booster
• ExUnit Booster
• GoTest Booster

Installation

gem install semaphore_test_boosters

What are Test Boosters

Test Boosters take your test suite and split the test files into multiple jobs. This allows you to quickly parallelize your test suite across multiple build machines.

As an example, let's take a look at the rspec_booster --job 1/10 command. It lists all the files that match the spec/**/*_spec.rb glob in your project, distributes them into 10 jobs, and execute the first job.

Split Configuration

Every test booster can load a split configuration file that helps the test booster to make a better distribution.

For example, if you have 3 RSpec Booster jobs, and you want to run:

• spec/a_spec.rb and spec/b_spec.rb in the first job
• spec/c_spec.rb and spec/d_spec.rb in the second job
• spec/e_spec.rb in the third job

you should put the following in your split configuration file:

[
  { "files": ["spec/a_spec.rb", "spec/b_spec.rb"] },
  { "files": ["spec/c_spec.rb", "spec/d_spec.rb"] },
  { "files": ["spec/e_spec.rb"] }
]

Semaphore uses Split configurations to split your test files based on their durations in the previous builds.

Leftover Files

Files that are part of your test suite, but are not in the split configuration file, are called "leftover files". These files will be distributed based on their file size in a round robin fashion across your jobs.

For example, if you have the following in your split configuration:

[
  { "files": ["spec/a_spec.rb"] }
  { "files": ["spec/b_spec.rb"] }
  { "files": ["spec/c_spec.rb"] }
]

and the following files in your spec directory:

# Files from split configuration ↓

spec/a_spec.rb
spec/b_spec.rb
spec/c_spec.rb

# Leftover files ↓

spec/d_spec.rb
spec/e_spec.rb

When you run the rspec_booster --job 1/3 command, the files from the configuration's first job and some leftover files will be executed.

rspec_booster --job 1/3

# => runs: bundle exec rspec spec/a_spec.rb spec/d_spec.rb

Booster will distribute your leftover files uniformly across jobs.

RSpec Booster

The rspec_booster loads all the files that match the spec/**/*_spec.rb pattern and uses the ~/rspec_split_configuration.json file to parallelize your test suite.

Example of running job 4 out of 32 jobs:

rspec_booster --job 4/32

Under the hood, the RSpec Booster uses the following command:

bundle exec rspec --format documentation --format json --out /home/<user>/rspec_report.json <file_list>

Optionally, you can pass additional RSpec flags with the TB_RSPEC_OPTIONS environment variable. You can also set a RSpec formatter with the TB_RSPEC_FORMATTER environment variable. Default formatter is documentation.

Example:

TB_RSPEC_OPTIONS='--fail-fast=3' TB_RSPEC_FORMATTER=Fivemat rspec_booster --job 4/32

# will execute:
bundle exec rspec --fail-fast=3 --format Fivemat --format json --out /home/<user>/rspec_report.json <file_list>

Cucumber Booster

The cucumber_booster loads all the files that match the features/**/*.feature pattern and uses the ~/cucumber_split_configuration.json file to parallelize your test suite.

Example of running job 4 out of 32 jobs:

cucumber_booster --job 4/32

Under the hood, the Cucumber Booster uses the following command:

bundle exec cucumber <file_list>

Minitest Booster

The minitest_booster loads all the files that match the test/**/*_test.rb pattern and uses the ~/minitest_split_configuration.json file to parallelize your test suite.

Example of running job 4 out of 32 jobs:

minitest_booster --job 4/32

Under the hood, the Minitest Booster uses the following command:

ruby -e 'ARGV.each { |f| require ".#{f}" }' <file_list>

ExUnit Booster

The ex_unit_booster loads all the files that match the test/**/*_test.exs pattern and uses the ~/ex_unit_split_configuration.json file to parallelize your test suite.

Example of running job 4 out of 32 jobs:

ex_unit_booster --job 4/32

Under the hood, the ExUnit Booster uses the following command:

mix test <file_list>

Go Test Booster

The go_test_booster loads all the files that match the **/*_test.go pattern and uses the ~/go_test_split_configuration.json file to parallelize your test suite.

Example of running job 4 out of 32 jobs:

go_test_booster --job 4/32

Under the hood, the Go Test Booster uses the following command:

go test <file_list>

Development

Integration testing

For integration tests we use test repositories that are located in https://github.com/renderedtext/test-boosters-tests.git.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/renderedtext/test-boosters.

License

The gem is available as open source under the terms of the MIT License.