(Automated) dependency compatibility tests

[schuderer]

As soon as we have some decent unit test coverage, we should try to find out the minimum versions of mllaunchpad's dependencies in order to be able to, in setup.cfg, explicitly specify which dependency versions work (no automatic <nextmajor, though, I'm against that, if we don’t also handle updates automatically).

Towards automatic handling of updates of dependencies, we need a mechanism that

1. Checks whether dependencies have had updates since the last check (on the first check, it treats every version as unseen, i.e. under test).
2. Runs all unit tests in a parametrized way against all (relevant*) versions of the dependencies (x supported Python versions). Look into a DOE-like setup which might save time here. Maybe Latin Hypercube is all we need. Also: cache packages locally.
3. Records which dep-versions have been tested, which fail and which succeed.
4. Reports this (alerting mllaunchpad team to incompatibilities that might have creeped in and could possibly be solved).
5. Adapts the setup.cfg's install_requires according to the compatibility table (we can programmatically access setup.cfg's settings, so maybe we can also easily change it without resorting to string replacements).

*) What the "relevant" versions of dependencies are, is a bit of a matter of opinion. We don't want to have to test all versions, but versions that matter (whatever that means). As a first suggestion, I'd say:
(a) the newest stable version on PyPI is always relevant, and
(b) the latest previous major version on PyPI is often relevant (e.g. we're on 1.1.20 and there's a 0.15.2 -- unless install_requires excludes previous major versions already), and
(c) two major versions back is usually not relevant any more.
Problem with this are of course non-semver dependencies, and/or dependencies where major feature additions happen across minor versions. So maybe we should take the extremes (newest version which hopefully works, but maybe doesn’t), and a version one (or two) major versions back which does not work, and from these extremes work in a pincer movement from both sides towards the edges of the interval of working versions. I wouldn’t go through patch versions, just major.minor.mostrecentpatch

Some stuff I brain-dumped into noxfile.py earlier:

# TODO: Auto-parametrize dependencies from setup.py (regular mega-dep-check):
# 1. loop over setup.cfg's "install_requires":
# 1a. get available versions of all dependencies
# pip install pandas==givemeallversions
# Could not find a version that satisfies the requirement pandas==givemeallversions (from versions: 0.1, 0.2b0, 0.2b1, 0.2, 0.3.0b0, 0.3.0b2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0rc1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc1, 0.8.0rc2, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0rc1, 0.19.0, 0.19.1, 0.19.2, 0.20.0rc1, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0rc1, 0.21.0, 0.21.1, 0.22.0, 0.23.0rc2, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0rc1, 0.24.0, 0.24.1, 0.24.2, 0.25.0rc0, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0rc0, 1.0.0, 1.0.1)
# 1b. get two newest major versions, or if only one, two newest minor versions
# 2. @nox.session(python=["3.6", my_py_ver])
#    @nox.parametrize('pandas', ['1.0.1', '0.25.3'])  # somehow do this in loop
#    ...
#    @nox.parametrize('flask', ['1.1.1', '0.12.5'])
#    def megatests(session, **dep_versions):
#        """Check recent versions of dependencies for compatibility"""
#        for dep, ver in dep_versions.items():
#           session.install(f'dep=={ver}')
#        ... install and carry out tests