Understand build/testing problems
Closed this issue · 5 comments
johann-petrak commented
johann-petrak commented
There are several issues here, possibly:
- setting up the dependencies takes a long time:
- for this we should use dependency caching inthe workflow
- pip struggles to resolve some dependencies and takes forever
- this sucks but the only way to get around this may be to find a working combination of pinned versions
- pinned versions are terrible as they force everyone else who wants to combine their code with gatenlp to use those versions as well, which may lead to conflicts
- but for just the build and tests run, we could use pinned versions in the github requirements only
johann-petrak commented
Helpful docs:
- pip https://pip.pypa.io/en/stable/topics/dependency-resolution/
- perhaps: use pipreqs: https://github.com/bndr/pipreqs to create requirements with pinned versions
- perhaps: use pip-tools https://github.com/jazzband/pip-tools
johann-petrak commented
Create outer new env, python 3.7
Test pipreqs
python3 -m venv env001
source env001/bin/activate
python -m pip install -u pip
pip install pipreqs
pipreqs
mv requirements.txt pipreqs-requirements.txt
Test pip-tools:
pip install pip-tools
cp requirements-github.txt pip-tools-requirements.txt
pip-compile pip-tools-requirements.txt
This shows that there are incompatibilities: importlib-metadata is required by 8 packages!
The problem seems to come from flake8 which requires a version less than 4.3 but ray needs version ge 4.8.3
There are also incompatibilities related to dill coming from tner vs allennlp vs multiprocess
johann-petrak commented
For now we should be able to live without testing tner, but flake8 is important.
See if we can just use pyflakes instead: by far not as useful, but lets use it for now!
Using the requirements file created from pip-tools, copied to requirements-github.txt
johann-petrak commented
This has been fixed now.