Explicit OpenCV flavor requirement conflicts with existing installs of OpenCV
faustomorales opened this issue · 20 comments
The current practice of specifying a requirement for opencv-python-headless causes conflicts for other installations of OpenCV. For example, if a user requires modules from opencv-contrib-python that are not available in opencv-python-headless but they install imgaug, pip will install both versions and thusly have conflicting versions of OpenCV with different features. To do this, I have to do:
pip install imgaug
pip uninstall -y opencv-python-headless
pip install opencv-contrib-python-headless
It seems this problem was contemplated in #324 so I suppose I'm just here to confirm those fears are well-founded.
From what I can tell, imgaug supports any of the four opencv-*-python-* flavors. In light of that, I humbly propose the following as options (with a preference for the first):
- Removing
opencv-*-*from theinstall_requiresinsetup.py. This is the simplest option and is compatible with the current installation instructions in the README which ask the user to install OpenCV on their own first. It is at this point that the user could choose the OpenCV flavor that makes sense for their application. - Implementing more logic to check for existing OpenCV versions. I think this will be kind of a pain but is what
scikit-imageseems to do and it may create problems for people who use package management systems likepipenvorpoetrywhere package installation sequence is ill-defined. It also will have sequencing issues for users who build their own customized versions ofopencv-pythonin order to use non-free algorithms in OpenCV.
I will gladly submit a PR for this if the relevant folks agree this is okay. Thanks!
Here's another interesting edge case to keep in mind:
I'm working from an NVidia Jetson Xavier with Jetpack 4.2.3. This comes preinstalled with opencv 3.3.1:
user@xavier-001:~$ python3 -c "import cv2; print(cv2.__version__)"
3.3.1However, trying to pip3 install imgaug gives this error:
ERROR: Could not find a version that satisfies the requirement opencv-python-headless (from imgaug) (from versions: none)
ERROR: No matching distribution found for opencv-python-headless (from imgaug)The workaround I used to install imgaug is as follows:
pip3 install the requirements of imgaug manually (from requirements.txt), except opencv-python-headless.
pip3 install --no-deps imgaug
This will probably become more and more of a nuisance when dealing with packages/modules depending on imgaug, as their installation will err or at least warn.
There is now a new release online in which setup.py accepts opencv-python, opencv-python-headless, opencv-contrib-python and opencv-contrib-python-headless -- if they are already installed. Otherwise it will fall back to opencv-python-headless as its dependency. This improves compatibility with libraries that use other opencv flavors than headless.
This change does not fix the issue that imgaug might end up being installed first, chooses its default opencv flavor (headless) and later on installed libraries are then incompatible with that flavor. At least in some cases that can be handled by installing the other libraries first and then in a separate command imgaug.
I'm rather reluctant currently to remove opencv from the dependencies list. For professional users that would probably be the most convenient choice. I suspect however that the majority of all users is rather inexperienced and would encounter problems if the library did not fully install itself, even if there was a seemingly clear error message. There is also the issue that removing the opencv dependency would probably break many existing builds that rely on opencv being auto-installed via imgaug.
I support an approach that helps inexperienced users. Experienced users can typically find workarounds like those suggested in this thread. I do wonder under what circumstances a user would rely on imgaug without also relying on OpenCV directly in their codebase, but that may just be a failure of my imagination. Bottom line, I'm okay closing this issue. Thanks for carefully considering the relevant tradeoffs.
As far as I can tell, the modification to setup.py works great when installing imgaug from source but doesn't help when installing the the pypi wheel. I'm guessing this is because the wheel METADATA for imgaug 0.4.0 has Requires-Dist: opencv-python.
Specifically, if I run pip install opencv-python-headless imgaug, I end up with both opencv-python and opencv-python-headless:
> pip install opencv-python-headless imgaug
Collecting opencv-python-headless
Using cached opencv_python_headless-4.2.0.32-cp36-cp36m-macosx_10_9_x86_64.whl (40.1 MB)
Collecting imgaug
Using cached imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
...
Collecting opencv-python
Using cached opencv_python-4.2.0.32-cp36-cp36m-macosx_10_9_x86_64.whl (47.9 MB)
...
Installing collected packages: numpy, opencv-python-headless, Shapely, Pillow, imageio, scipy, opencv-python, pyparsing, six, cycler, kiwisolver, python-dateutil, matplotlib, PyWavelets, decorator, networkx, scikit-image, imgaug
Successfully installed Pillow-7.0.0 PyWavelets-1.1.1 Shapely-1.7.0 cycler-0.10.0 decorator-4.4.1 imageio-2.6.1 imgaug-0.4.0 kiwisolver-1.1.0 matplotlib-3.1.3 networkx-2.4 numpy-1.18.1 opencv-python-4.2.0.32 opencv-python-headless-4.2.0.32 pyparsing-2.4.6 python-dateutil-2.8.1 scikit-image-0.16.2 scipy-1.4.1 six-1.14.0
I couldn't find a way list alternative dependencies in a wheel, and it appears the idea was considered and rejected in PEP 426.
Doesn't a virtual env resolve the issue?
Doesn't a virtual env resolve the issue?
No, this doesn't give us a way to pip install imgaug==0.4 without it installing opencv-python (which we don't want). We've pinned our package to require imgaug==0.3.0 so that we can use opencv-python-headless.
I have also encountered this issue where we are trying to break the dependency on opencv-python (i.e. the non-headless version), but it makes its way in due to albumentations==0.4.6 -> imgaug==0.4.0.
I'm no Python packaging expert, but It seems like the wheel was made in an environment where opencv-python was installed so it made its way into the wheel dependencies. Can we get a new wheel with opencv-python-headless, please?
Does anyone know if pyproject.toml (replacing setup.py) could help us, or is this really a wheel-building issue?
I'm no Python packaging expert, but It seems like the wheel was made in an environment where
opencv-pythonwas installed so it made its way into the wheel dependencies. Can we get a new wheel withopencv-python-headless, please?
I second that.
As a workaround (besides holding at 0.3.0) one can simply avoid the binary/wheel method for imgaug, as in:
pip install --no-binary imgaug imgaug # ... other deps
...or in a requirements.txt file...
# ... other deps
imgaug --no-binary imgaug
# ... other depsAnd there is yet another option: I have created a pseudo-package re-branding opencv-python-headless as opencv-python (and trying to re-export its install-time version number, too). Thus, you could install this package prior to installing some dependency chain to avoid dragging in opencv-python proper (and thereby replacing headless).
I wonder, though, why the OpenCV Python bindings did not decide to solve the contrib and X11 options via the extras mechanism. So packages which depend on these features could depend on opencv-python[contrib] or opencv[contrib,x11], or in turn make them conditional in extras of their own (e.g. extras_require={'plot':['opencv-python[x11]']})...
Just spent a whole morning getting bit by this. There should at least be a section in the README about how to install with different opencv flavors, and even better an officially supported way of making it happen.
@aleju, I can also confirm that the uploaded wheel triggers the opencv-python dependency (as opposed to opencv-python-headless). I think this could be resolved by either of the following two things:
- re-upload the wheel, making sure that the correct library (
opencv-python-headless) is referenced in it - do a new release and then upload again
Would that be feasible?
pip install --upgrade pip
Have a try.
Can you quickly elaborate what changed?
- re-upload the wheel, making sure that the correct library (
opencv-python-headless) is referenced in it
IINM, that could be as simple as:
pip download imgaug
wheel unpack imgaug*.whl
sed -i s/opencv-python/opencv-python-headless/ imgaug-0.4.0/imgaug-0.4.0.dist-info/METADATA
wheel pack --build-number 2 imgaug-0.4.0
twine upload imgaug-0.4.0-2-py2.py3-none-any.whl
Anyone with the credentials for imgaug on PyPI could do that.
- do a new release and then upload again
Yes, a true post-release according to PEP 440 (like 0.4.0.post1) would probably be better.
On Windows, this creates another interesting error if you are trying to install imgaug while having an open Python process that is using OpenCV.
ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: <REDACTED>\\.venv\\Lib\\site-packages\\cv2\\cv2.cp38-win_amd64.pyd'
Consider using the `--user` option or check the permissions.I worked around this using --no-dependencies.
@patzm Could you, being German, try to write/call @aleju ? :) Here are his contact details: http://www.ajung.name/impressum.html
It seems like @aleju stopped working on GitHub around June last year. If he doesn't intend to (exclusively) continue maintaining this repo, maybe someone can offer a helping hand 🤔. I opened #764 for that matter. Let's discuss over there. If Alex doesn't respond in say a month, I can for sure give him a call 😉.
Yo apparently the recent release v1.0.0 of albumentations avoids this issue. Check out the release notes: https://github.com/albumentations-team/albumentations/releases/tag/1.0.0
Yo apparently the recent release
v1.0.0ofalbumentationsavoids this issue. Check out the release notes: https://github.com/albumentations-team/albumentations/releases/tag/1.0.0
In some projects (e.g. paddleOCR), there is imgaug in the "requirements.txt" file. Is there any convenient method to replace it with albumentations?
@patzm