v0.20 - Refactor to Python

[finalduty]

Overview

Reviewing this repo after a few years has found that while it is popular, I have not kept up with development of it as much as I would like.

After review, the projects goal to provide an easy to use single-file executable with zero external requirements remains intact but it is becoming increasingly obvious that using bash is not the most effective language going forward.

The repository health shows some concerning symptoms that make me think this contribution is not as good as it should be. Notably, the age of issues is very high and the ratio of forks to pull requests suggest that the community find it easier to adapt/fix things themselves than push back to this repo. This leads to a dilution of community efforts, duplication of efforts and reduction in potential benefits gained by using this repo.

Problem/Issues

1. A lot (most) of the issues raised relate to individual tests having problems that are difficult to test/capture. Additionally there is currently no way of easily detecting/preventing regressions.
2. Community contribution experience is poor. Issues remain open and PRs unaddressed for long periods of time
3. Ability to support different / future versions of the benchmarks is difficult with the current structure

Solutions

1. Implement testing suite and automated testing

   • Implement unit tests using Pytest
   • Implement GitHub Actions to run test suites

2. Contribution experience
   Many of the issues that have remained open are due to difficulty in fixing/replicating/preventing-regression. Automated regression testing as above will help, but there is still a process improvement to be made. This may need to be split to a different issue.

   • Community Standards
     • Add CONTRIBUTING.md (https://docs.github.com/articles/setting-guidelines-for-repository-contributors/)
     • Add Issue Template
     • Add Pull Request Template
   • Styling
     • Implement and ensure conformance with flake8
     • Implement and ensure code styling with black
     • Provide documentation on logical flow / concepts in place - Possibly a section of CONTRIBUTING.md?

3. Improve support for various benchmarks/os/python versions

   • Rename test functions to be more descriptive / not tied to a specific benchmark version, e.g. test_1.1.1.x -> test_kernel_module_is_disabled
   • Add ability to support multiple benchmark versions
   • Add ability to support multiple OS/Python versions with tox

Supercedes #28