Version conflict in docker
Closed this issue · 7 comments
beckstefan commented
This is the full traceback when calling in latest docker ocrd/all:maximum
Traceback (most recent call last):
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 567, in _build_master
ws.require(__requires__)
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 884, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 775, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (numpy 1.19.2 (/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages), Requirement.parse('numpy<1.19.0,>=1.16.0'), {'tensorflow'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/local/sub-venv/headless-tf22/bin/ocrd-anybaseocr-crop", line 33, in <module>
sys.exit(load_entry_point('ocrd-anybaseocr', 'console_scripts', 'ocrd-anybaseocr-crop')())
File "/usr/local/local/sub-venv/headless-tf22/bin/ocrd-anybaseocr-crop", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 105, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/build/ocrd_anybaseocr/ocrd_anybaseocr/cli/ocrd_anybaseocr_cropping.py", line 42, in <module>
from ocrd.decorators import ocrd_cli_options, ocrd_cli_wrap_processor
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd/__init__.py", line 17, in <module>
from ocrd.processor.base import run_processor, run_cli, Processor
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd/processor/__init__.py", line 1, in <module>
from .base import (
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd/processor/base.py", line 9, in <module>
from ocrd_utils import VERSION as OCRD_VERSION, MIMETYPE_PAGE
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd_utils/__init__.py", line 71, in <module>
from .constants import (
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/ocrd_utils/constants.py", line 4, in <module>
from pkg_resources import get_distribution
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3238, in <module>
@_call_aside
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3222, in _call_aside
f(*args, **kwargs)
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3251, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 569, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 582, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages/pkg_resources/__init__.py", line 775, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (numpy 1.19.2 (/usr/local/local/sub-venv/headless-tf22/lib/python3.6/site-packages), Requirement.parse('numpy<1.19.0,>=1.16.0'), {'tensorflow'})
kba commented
numpy is required in a few projects
cor-asv-ann/requirements.txt
cor-asv-fst/requirements.txt
core/ocrd_utils/requirements.txt
dinglehopper/requirements.txt
ocrd_anybaseocr/requirements.txt
ocrd_calamari/requirements.txt
ocrd_keraslm/requirements.txt
ocrd_segment/requirements.txt
ocrd_wrap/requirements.txt
But none of them define upper-bounds for the version. Would introducing numpy < 1.19.0 in core fix this issue? It's a workaround but if that's what it takes to get anybaseocr running again @bertsky @stweil ?
Why did this happen now? Did tensorflow 2.2 change it's requirements?
stweil commented
My latest local installation has these numpy versions:
venv-20200925/local/sub-venv/headless-tf1/lib/python3.7/site-packages/numpy-1.19.2.dist-info
venv-20200925/local/sub-venv/headless-tf21/lib/python3.7/site-packages/numpy-1.18.5.dist-info
venv-20200925/local/sub-venv/headless-tf22/lib/python3.7/site-packages/numpy-1.19.2.dist-info
venv-20200925/local/sub-venv/headless-torch14/lib/python3.7/site-packages/numpy-1.19.2.dist-info
ocrd_pc_segmentation already has a hard requirement for numpy==1.18.5.
stweil commented
I just checked the TF versions and was surprised to see the result:
venv-20200925/local/sub-venv/headless-tf21/lib/python3.7/site-packages/tensorflow-2.3.1.dist-info
venv-20200925/local/sub-venv/headless-tf22/lib/python3.7/site-packages/tensorflow-2.1.2.dist-info
So tf21 uses the latest TensorFlow 2.3 while tf22 uses TensorFlow 2.1. That's strange and unexpected.
stweil commented
All versions of TensorFlow seem to have identical requirements for numpy:
grep numpy venv-20200925/local/sub-venv/headless-tf*/lib/python3.7/site-packages/tensorflow*.dist-info/METADATA
venv-20200925/local/sub-venv/headless-tf1/lib/python3.7/site-packages/tensorflow_gpu-1.15.4.dist-info/METADATA:Requires-Dist: numpy (<1.19.0,>=1.16.0)
venv-20200925/local/sub-venv/headless-tf21/lib/python3.7/site-packages/tensorflow-2.3.1.dist-info/METADATA:Requires-Dist: numpy (<1.19.0,>=1.16.0)
venv-20200925/local/sub-venv/headless-tf22/lib/python3.7/site-packages/tensorflow-2.1.2.dist-info/METADATA:Requires-Dist: numpy (<1.19.0,>=1.16.0)
stweil commented
Would introducing numpy < 1.19.0 in core fix this issue?
It looks like that would be a working solution for all sub venvs with TensorFlow, although it unnecessarily enforces older versions of numpy also for the venvs without TensorFlow.
A side note: did I ever say that TensorFlow is a nightmare?
bertsky commented
So this has become OCR-D/core#620.
bertsky commented
BTW, here's a temporary workaround for existing Docker images or native venvs:
cd /usr/local # only in Docker installation
. venv/bin/activate && cd venv # only in native installation
pip install "numpy<1.19"
. local/sub-venv/headless-tf1/bin/activate && pip install "numpy<1.19"
. local/sub-venv/headless-tf21/bin/activate && pip install "numpy<1.19"
. local/sub-venv/headless-tf22/bin/activate && pip install "numpy<1.19"