Use mutation testing to improve (unit) tests

[ericcornelissen]

Engineering Initiative

Description

With #121 merged the packages can now be mutation tested. This issue is to keep track of which packages have been updated based on the result of running mutation tests.

The goal of mutation testing is to improve (unit) tests to actually verify the implementation, rather than just provide coverage. Mutation testing achieves this by mutating the source code and running the tests to see if the change is detected. If the change was not detected it is saved as a "mutant" and reported back to the user. If no mutants are found, that implies changes to the code base will be caught by the tests.

Progress

• packages/benchmarking (#212)
• packages/cli (#249)
• packages/core (#317)
• packages/language-css (#197)
• packages/language-html (#269)
• packages/language-js (#227)
• packages/language-utils (#218)
• packages/mangler-css-classes (#260)
• packages/mangler-css-variables (#193)
• packages/mangler-html-attributes (#267)
• packages/mangler-html-ids (#216)
• packages/mangler-utils (#196)
• packages/testing (#253)
• packages/types (no tests)

Followup

• Make the get-filters.js filter for mutation testing dynamic (#324)

Related Issues & Pull Requests

• #113
• #117
• #121
• #134
• #140
• #142
• #153
• #193
• #196
• #197
• #204
• #212
• #216
• #218
• #227
• #249
• #253
• #260
• #267
• #269
• #288
• #317
• #385

[ericcornelissen]

If #204 is merged, which restricts mutation tests to unit tests, the list should be updated as various checked packages are not quite done yet. In particular:

• packages/benchmarking: Doesn't have dedicated unit tests yet (see also #113)
• language/html: Doesn't have dedicated unit tests yet (see also #113)
• language/js: Doesn't meet the 50% threshold using only unit tests

EDIT: Updated the issue description accordingly.

[ericcornelissen]

Consider replacing mutation report artifects by using Stryker Dashboard (documentation). Note the following line from the documentation:

MODULE: Optional. If you want to store multiple reports for a version, you can use this value to separate them logically, for example, in a mono-repo setup where each package (or project or module) delivers a report.

EDIT: Done in #288