/testing-in-bash

Bash test framework comparison

Primary LanguageShellGNU General Public License v2.0GPL-2.0

In search of the best test framework for Bash

There are many testing frameworks for Bash. However, many of them are not actively maintained, or are used only by a small group of people.

In this comparision I've included well-established and new testing frameworks that had at least recent release and have more than 50 Stars on GitHub:

  • bashunit, a fast, simple and flexible testing library for your bash scripts
  • Bats, a mature testing library with a bit of a custom syntax for more concise tests
  • shUnit2, possibly the oldest Bash test framework around, supporting test files in pure Bash
  • bash_unit, a new and featureful test framework where test cases can be written in pure Bash
  • ShellSpec, a BDD style testing framework with its own DSL
  • shpec, a BDD style testing framework similar to RSpec, Jasmine, and mocha

Test drive

To start fiddling with any of the test projects just run the install.sh to download and extract the particular testing framework to a local directory, then execute test.sh to start the suite.

Detailed comparision

bashunit Bats shUnit2 bash_unit ShellSpec shpec
Discover and run all tests
in the project		 ✔️ ✔️ ❌ ? ⚠ ? ✔️ ? ✔️
Run subset of tests or
a specific test		 ✔️ ✔️ ? ⚠ ? ✔️ ✔️ ❌ issue
Parallel test execution ❌ ✔️ ❌ ❌ ✔️ ❌
Skip or ignore tests ✔️ ✔️ ⚠ ? ✔️ ? ✔️ ❌ issue
Rich assertion library ✔️ ⚠ ? ⚠ ? ⚠ ? ✔️ ? ✔️ ?
Extensible with custom assertions ✔️ ✔️ ? ✔️ ? ✔️ ? ✔️ ? ✔️ ?
Before / After ✔️ ✔️ ⚠ issue ✔️ ✔️ ❌ issue
BeforeAll / AfterAll ✔️ ❌ issue ⚠ issue ✔️ ✔️ ? ❌ issue
Parameterized tests
and test data		 ✔️ ❌ issue ❌ ❌ ✔️ ❌
Mocking * ⚠ issue ✔️ ? ⚠ ? ✔️ ? ✔️ ? ✔️ ?
Isolation between tests ** ✔️ ✔️ doc ❌ ✔️ ✔️
Nice local report ✔️ ⚠ ? ✔️ ✔️ ✔️ ✔️
CI compatible report ⚠ ✔️ TAP ❌ issue ✔️ TAP ✔️ TAP ❌
Zero Dependency and
implemented in Bash ***		 ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Easy to install **** ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
Tests written in Bash ✔️ ⚠ almost ✔️ ✔️ ❌ ? ❌ ?
Compatible with the
Unofficial Bash Strict Mode *****		 ⚠ ✔️ ⚠ ? ✔️ ⚠ issue ✔️
Maintained ✔️ ⚠ ? ⚠ ? ✔️ ? ✔️ ? ⚠ ?
Mature ❌ since 2023 ✔️ since 2011 ✔️ since 2007 ✔️ since 2016 ❌ since 2019 ✔️ since 2013
Good documentation ✔️ ? ⚠ ? ✔️ ? ✔️ ? ✔️ ? ✔️ ?

* Mocking:
The testing framework is compatible with the common mocking techniques.

** Isolation between tests:
Aliases, custom function definitions, Bash options and variables are not leaking from one test to another.

*** Zero Dependency and implemented in Bash:
Apart from a few common tools like coreutils and grep, you don't need anything to use the framework.
The source code of the framework is itself implemented as Bash scripts. If you are comfortable with Bash, you can read and even modify it.

**** Easy to install:
Many of the frameworks support more than one alternatives on how to set it up quickly. It's possible to install it with your favourite package manager and some of them even have an official Docker image. Personally, I try to avoid installing project dependencies globally as much as I can, so I just decided to grab a tar.gz file with the latest release from GitHub.

***** Compatible with the Unofficial Bash Strict Mode:
Sourcing a script file that use the Unofficial Bash Strict Mode should not break the test framework. Of course, custom assertions and the test code should also work with strict mode but that's up for the user. Also, the test framework shouldn't set Bash options as well to ensure that executing sourced functions work similarly to when they are not sourced.

Related posts, other comparisions

Contribution guide

If you know a testing framework that should be mentioned here or find that there's room for improvement feel free to open an issue or Pull Request. (See CONTRIBUTING.md.)