Broken Docker image
c0c0n3 opened this issue · 3 comments
Describe the bug
Can't build the Docker image at the moment. The step where we install the Python deps fails. Since we don't pin every dep, most likely we've bumped in another instance of self-inflicted dependency hell.
To Reproduce
See e.g. PR #730's build log
Or run docker build -t orchestracities/quantumleap .
from the root dir yourself.
Expected behavior
You should be able to build the Docker image.
Environment
- CircleCI/Linux
- MacOS/Aarch64
- MacOS/x86-64
Additional context
This is just another installment of the neverending dependency hell saga, see e.g. #273.
So we should really move away from Python 3.8, but that's too much effort. An avenue we can explore is to upgrade pyyaml
from version 5
to version 6
. With this upgrade I could make some progress locally, it remains to be seen if that works on CircleCI too. (I still need to upgrade other deps though.) The reason we can't upgrade to version 6
without pain is PR #516 which added a ~=5.4
constraint to the Pipenv dev deps
But I think that was a useless thing to do anyway. In fact, the same PR installs the latest pyyaml
version on every CircleCI test session run:
which means at the moment we get version 6
on CircleCI. So what's the use of adding the ~=5.4
constraint to the Pipenv dev deps? Also the main section of the Pipenv file requires pyyaml >= 4.2
. So it looks like something fell through the cracks here. My gut is that we should undo this change
and pin pyyaml
to version 6
in the main section of Pipfile.
Notice that the upgrade path isn't straightforward. The problem is that, like us, other Pythonistas out there also have dependency hell issues. Here's an example. If you delete the Pipenv lock file and try regenerating it (or just upgrade deps) you'll see that all the QL tests will break. One of the errors is this
ImportError: cannot import name 'soft_unicode' from 'markupsafe'
(see pallets/markupsafe#282)
Why is that happening? Well, our deps declaration in the Pipfile lead to the following dependency tree---just showing the paths that contain the markupsafe
package.
connexion 2.14.2 Connexion - API first applications with OpenAPI/Swagger and Flask
├── flask >=1.0.4,<2.3
| ├── jinja2 >=2.10.1,<3.0
| └── markupsafe >=0.23
├── swagger-ui-bundle >=0.0.2,<0.1
└── jinja2 >=2.0
└── markupsafe >=0.23
flask 1.1.4 A simple framework for building complex web applications.
├── jinja2 >=2.10.1,<3.0
└── markupsafe >=0.23
pytest-flask 1.2.0 A set of py.test fixtures to test Flask applications.
├── flask *
├── jinja2 >=2.10.1,<3.0
└── markupsafe >=0.23
With these dependency constraints, the dependency solver can pick any version of the jinja2
package greater or equal to 2.10.1
but less than 3
and any markupsafe
version greater or equal to 0.23
. So the solver picks
jinja2 2.11.3
markupsafe 2.1.3
But in actual fact, jinja2 2.11.3
is incompatible with markupsafe 2.1.3
because it tries to import soft_unicode
from markupsafe
but soft_unicode
isn't in markupsafe 2.1.3
. The last version containing soft_unicode
is markupsafe 2.0.1
. So the jinja2
maintainers should've specified an upper version bound of 2.0.1
for markupsafe
in their deps, instead of just a lower bound of 0.23
:
Anyways, what can we do about it? One option would be to add an explicit dependency on markupsafe 2.0.1
in our Pipfile. This isn't great for all the obvious reasons, but it's an option nonetheless. Another option would be to upgrade connexion
and flask
to recent versions which depend on jinja2 > 3.0
---recent jinja2
versions actually work with the latestmarkupsafe
version, 2.1.3.
But Python being that wonderful dynamic/untyped soup that it is, I'm a bit scared of upgrading flask
without extensive testing.