Duplicated lists of dependencies
Opened this issue · 2 comments
Our requirements.txt
is mainly used as a lock / constraints file. We use it everywhere with pip install -c requirements.txt
(except in the docker file, see PR).
pip-compile
is not able to generate a constraints file from all mentioned dependencies in pyproject.toml
(main + extras).
pip
does not support extras dependencies in constraints files (that's why we have strip-extras = true
), and does not support hashes when installing the package locally in editable mode (that's why we have generate-hashes = false
). We need editable mode to be able to make changes without having to reinstal the package between two tests runs for example.
In #3359, we took this approach:
- main and extra dependencies listed in
pyproject.toml
- all sort of dependencies manually combined in
requirements.in
requirements.txt
generated usingpip-compile
.
Ideally we would to avoid this duplication, and have one source of truth for all kinds of dependencies. Then have a constraints/lock file for reproducible builds/tests/installations.
Solution #0: status quo
This duplication is fine. The list of package does not change so often.
Solution #1: Get rid of extra dependencies
Simplify everything, and install all dependencies instead of having monitoring
, postgresql
, memcached
extras.
We would still need a distinction for development dependencies, but that's straightforward and doesn't really require pinning.
Solution #2: split into several *.in
files
setuptools
is able to read *.in
files from pyproject.toml
. That's what we do in kinto-attachment
We could have:
requirements.in
requirements-postgresql.in
requirements-memcached.in
requirements-monitoring.in
requirements-test.in
requirements-dev.in
The source of truth for the list of dependencies would be these .in
files.
Solution #3: use poetry
poetry
has a simple .Piplock
feature, and has many features around fine-grained installation of extra dependencies and groups.
But poetry
does not support the [project]
section (yet).
We would have to move away from setuptools and setuptools-scm, and use a [tools.poetry]
section.
I'm a big fan of solution 1, keep things as simple as possible. We only publish one package anyway, right? If anybody wants to make a custom build it should still be just as simple to keep doing that.
Something to consider, with Solution 1
we can't just use pip install kinto && kinto start
to start a quick in-memory storage backend. Because we would require postgresql client libs (ie. compiling psycopg for example). Also in all our repos' CI like kinto-wizard, etc.