No usable version of opencv for Python 3.9
wvxvw opened this issue · 12 comments
Hello,
I'm trying to install opencv from conda-forge in a matrix build in this GitHub Actions script: https://github.com/drcandacemakedamoore/cleanX/blob/wvxvw/matrix-build/.github/workflows/on-commit.yml#L84 . It's building a package and then runs tests, which fail due to import cv2 failing.
If I try the latest version, the failure looks like this:
import cv2
E ImportError: dlopen(/usr/local/miniconda/envs/cleanx-build/lib/python3.9/site-packages/cv2.cpython-39-darwin.so, 2): Symbol not found: _mp_get_memory_functions
E Referenced from: /usr/local/miniconda/envs/cleanx-build/lib/libgnutls.30.dylib
E Expected in: /usr/local/miniconda/envs/cleanx-build/lib/libhogweed.4.dylib
E in /usr/local/miniconda/envs/cleanx-build/lib/libgnutls.30.dylib
This happens with both opencv and opencv<4.5.0, then I try opencv<4.4.0, then that's not installable on Python 3.9.
There's similar problem on Linux, but the missing symbol is different:
import cv2
E ImportError: /usr/share/miniconda/envs/cleanx-build/lib/python3.9/site-packages/../../././libgnutls.so.30: undefined symbol: mpn_add_1, version HOGWEED_4
Bottom line, the two versions of opencv I found I could try to install with Python 3.9 are broken on Mac and Linux. Are there any versions that do work? Is this a misconfigured dependency?
Please don't take this the wrong way, but we ask for some more information than the error you've provided. The information requested in the original issue template is critical to us being able to help you.
Please click "open new issue" to review the requested information and please try to provide it here.
For what its worth, I install opencv every day using conda-forge on many Ubuntu machines. Albeit they are 20.04 but I think some people on my team were on 18.04 without issue for a long time.
09:08 $ conda create --name opencv python=3.9 opencv --yes
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/mark/mambaforge/envs/opencv
added / updated specs:
- opencv
- python=3.9
The following packages will be downloaded:
package | build
---------------------------|-----------------
harfbuzz-2.8.2 | h83ec7ef_0 2.0 MB conda-forge
hdf5-1.10.6 |nompi_h3c11f04_101 3.0 MB conda-forge
libblas-3.9.0 | 8_openblas 11 KB conda-forge
libcblas-3.9.0 | 8_openblas 11 KB conda-forge
libgcc-ng-11.1.0 | hc902ee8_2 6.7 MB conda-forge
libgfortran-ng-7.5.0 | h14aa051_19 22 KB conda-forge
libgfortran4-7.5.0 | h14aa051_19 1.3 MB conda-forge
libgomp-11.1.0 | hc902ee8_2 427 KB conda-forge
liblapack-3.9.0 | 8_openblas 11 KB conda-forge
liblapacke-3.9.0 | 8_openblas 11 KB conda-forge
libopenblas-0.3.12 |pthreads_hb3c22a3_1 8.2 MB conda-forge
libopencv-4.5.2 | py39h70bf20d_1 34.1 MB conda-forge
libstdcxx-ng-11.1.0 | h56837e0_2 4.2 MB conda-forge
numpy-1.21.1 | py39hdbf815f_0 6.2 MB conda-forge
opencv-4.5.2 | py39hf3d152e_1 21 KB conda-forge
pip-21.2.1 | pyhd8ed1ab_0 1.1 MB conda-forge
py-opencv-4.5.2 | py39hef51801_1 1.1 MB conda-forge
python-3.9.6 |h49503c6_1_cpython 27.5 MB conda-forge
tzdata-2021a | he74cb21_1 121 KB conda-forge
------------------------------------------------------------
Total: 96.1 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
ca-certificates conda-forge/linux-64::ca-certificates-2021.5.30-ha878542_0
cairo conda-forge/linux-64::cairo-1.16.0-h6cf1ce9_1008
certifi conda-forge/linux-64::certifi-2021.5.30-py39hf3d152e_0
dbus conda-forge/linux-64::dbus-1.13.6-h48d8840_2
expat conda-forge/linux-64::expat-2.4.1-h9c3ff4c_0
ffmpeg conda-forge/linux-64::ffmpeg-4.3.1-hca11adc_2
fontconfig conda-forge/linux-64::fontconfig-2.13.1-hba837de_1005
freetype conda-forge/linux-64::freetype-2.10.4-h0708190_1
gettext conda-forge/linux-64::gettext-0.19.8.1-h0b5b191_1005
glib conda-forge/linux-64::glib-2.68.3-h9c3ff4c_0
glib-tools conda-forge/linux-64::glib-tools-2.68.3-h9c3ff4c_0
gmp conda-forge/linux-64::gmp-6.2.1-h58526e2_0
gnutls conda-forge/linux-64::gnutls-3.6.13-h85f3911_1
graphite2 conda-forge/linux-64::graphite2-1.3.13-h58526e2_1001
gst-plugins-base conda-forge/linux-64::gst-plugins-base-1.14.5-h0935bb2_2
gstreamer conda-forge/linux-64::gstreamer-1.18.4-h76c114f_2
harfbuzz conda-forge/linux-64::harfbuzz-2.8.2-h83ec7ef_0
hdf5 conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_101
icu conda-forge/linux-64::icu-68.1-h58526e2_0
jasper conda-forge/linux-64::jasper-1.900.1-h07fcdf6_1006
jbig conda-forge/linux-64::jbig-2.1-h7f98852_2003
jpeg conda-forge/linux-64::jpeg-9d-h36c2ea0_0
krb5 conda-forge/linux-64::krb5-1.17.2-h926e7f8_0
lame conda-forge/linux-64::lame-3.100-h7f98852_1001
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_1
lerc conda-forge/linux-64::lerc-2.2.1-h9c3ff4c_0
libblas conda-forge/linux-64::libblas-3.9.0-8_openblas
libcblas conda-forge/linux-64::libcblas-3.9.0-8_openblas
libclang conda-forge/linux-64::libclang-11.1.0-default_ha53f305_1
libdeflate conda-forge/linux-64::libdeflate-1.7-h7f98852_5
libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libevent conda-forge/linux-64::libevent-2.1.10-hcdb4288_3
libffi conda-forge/linux-64::libffi-3.3-h58526e2_2
libgcc-ng conda-forge/linux-64::libgcc-ng-11.1.0-hc902ee8_2
libgfortran-ng conda-forge/linux-64::libgfortran-ng-7.5.0-h14aa051_19
libgfortran4 conda-forge/linux-64::libgfortran4-7.5.0-h14aa051_19
libglib conda-forge/linux-64::libglib-2.68.3-h3e27bee_0
libgomp conda-forge/linux-64::libgomp-11.1.0-hc902ee8_2
libiconv conda-forge/linux-64::libiconv-1.16-h516909a_0
liblapack conda-forge/linux-64::liblapack-3.9.0-8_openblas
liblapacke conda-forge/linux-64::liblapacke-3.9.0-8_openblas
libllvm11 conda-forge/linux-64::libllvm11-11.1.0-hf817b99_2
libopenblas conda-forge/linux-64::libopenblas-0.3.12-pthreads_hb3c22a3_1
libopencv conda-forge/linux-64::libopencv-4.5.2-py39h70bf20d_1
libpng conda-forge/linux-64::libpng-1.6.37-h21135ba_2
libpq ramonaoptics/linux-64::libpq-12.3-h5957347_100
libprotobuf conda-forge/linux-64::libprotobuf-3.16.0-h780b84a_0
libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-11.1.0-h56837e0_2
libtiff conda-forge/linux-64::libtiff-4.3.0-hf544144_1
libuuid conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
libwebp-base conda-forge/linux-64::libwebp-base-1.2.0-h7f98852_2
libxcb conda-forge/linux-64::libxcb-1.13-h7f98852_1003
libxkbcommon conda-forge/linux-64::libxkbcommon-1.0.3-he3ba5ed_0
libxml2 conda-forge/linux-64::libxml2-2.9.12-h72842e0_0
lz4-c conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_0
mysql-common conda-forge/linux-64::mysql-common-8.0.25-ha770c72_2
mysql-libs conda-forge/linux-64::mysql-libs-8.0.25-hfa10184_2
ncurses conda-forge/linux-64::ncurses-6.2-h58526e2_4
nettle conda-forge/linux-64::nettle-3.6-he412f7d_0
nspr conda-forge/linux-64::nspr-4.30-h9c3ff4c_0
nss conda-forge/linux-64::nss-3.67-hb5efdd6_0
numpy conda-forge/linux-64::numpy-1.21.1-py39hdbf815f_0
opencv conda-forge/linux-64::opencv-4.5.2-py39hf3d152e_1
openh264 conda-forge/linux-64::openh264-2.1.1-h780b84a_0
openssl conda-forge/linux-64::openssl-1.1.1k-h7f98852_0
pcre conda-forge/linux-64::pcre-8.45-h9c3ff4c_0
pip conda-forge/noarch::pip-21.2.1-pyhd8ed1ab_0
pixman conda-forge/linux-64::pixman-0.40.0-h36c2ea0_0
pthread-stubs conda-forge/linux-64::pthread-stubs-0.4-h36c2ea0_1001
py-opencv conda-forge/linux-64::py-opencv-4.5.2-py39hef51801_1
python conda-forge/linux-64::python-3.9.6-h49503c6_1_cpython
python_abi conda-forge/linux-64::python_abi-3.9-2_cp39
qt conda-forge/linux-64::qt-5.12.9-h9d6b050_2
readline conda-forge/linux-64::readline-8.1-h46c0cb4_0
setuptools conda-forge/linux-64::setuptools-49.6.0-py39hf3d152e_3
sqlite conda-forge/linux-64::sqlite-3.36.0-h9cd32fc_0
tk conda-forge/linux-64::tk-8.6.10-h21135ba_1
tzdata conda-forge/noarch::tzdata-2021a-he74cb21_1
wheel conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
x264 conda-forge/linux-64::x264-1!161.3030-h7f98852_1
xorg-kbproto conda-forge/linux-64::xorg-kbproto-1.0.7-h7f98852_1002
xorg-libice conda-forge/linux-64::xorg-libice-1.0.10-h7f98852_0
xorg-libsm conda-forge/linux-64::xorg-libsm-1.2.3-hd9c2040_1000
xorg-libx11 conda-forge/linux-64::xorg-libx11-1.7.2-h7f98852_0
xorg-libxau conda-forge/linux-64::xorg-libxau-1.0.9-h7f98852_0
xorg-libxdmcp conda-forge/linux-64::xorg-libxdmcp-1.1.3-h7f98852_0
xorg-libxext conda-forge/linux-64::xorg-libxext-1.3.4-h7f98852_1
xorg-libxrender conda-forge/linux-64::xorg-libxrender-0.9.10-h7f98852_1003
xorg-renderproto conda-forge/linux-64::xorg-renderproto-0.11.1-h7f98852_1002
xorg-xextproto conda-forge/linux-64::xorg-xextproto-7.3.0-h7f98852_1002
xorg-xproto conda-forge/linux-64::xorg-xproto-7.0.31-h7f98852_1007
xz conda-forge/linux-64::xz-5.2.5-h516909a_1
zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1010
zstd conda-forge/linux-64::zstd-1.5.0-ha95c52a_0
Downloading and Extracting Packages
libgfortran4-7.5.0 | 1.3 MB | ######################################################################### | 100%
python-3.9.6 | 27.5 MB | ######################################################################### | 100%
numpy-1.21.1 | 6.2 MB | ######################################################################### | 100%
libblas-3.9.0 | 11 KB | ######################################################################### | 100%
libcblas-3.9.0 | 11 KB | ######################################################################### | 100%
harfbuzz-2.8.2 | 2.0 MB | ######################################################################### | 100%
py-opencv-4.5.2 | 1.1 MB | ######################################################################### | 100%
liblapack-3.9.0 | 11 KB | ######################################################################### | 100%
pip-21.2.1 | 1.1 MB | ######################################################################### | 100%
libgomp-11.1.0 | 427 KB | ######################################################################### | 100%
libopencv-4.5.2 | 34.1 MB | ######################################################################### | 100%
libgcc-ng-11.1.0 | 6.7 MB | ######################################################################### | 100%
tzdata-2021a | 121 KB | ######################################################################### | 100%
liblapacke-3.9.0 | 11 KB | ######################################################################### | 100%
libopenblas-0.3.12 | 8.2 MB | ######################################################################### | 100%
libgfortran-ng-7.5.0 | 22 KB | ######################################################################### | 100%
hdf5-1.10.6 | 3.0 MB | ######################################################################### | 100%
opencv-4.5.2 | 21 KB | ######################################################################### | 100%
libstdcxx-ng-11.1.0 | 4.2 MB | ######################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate opencv
#
# To deactivate an active environment, use
#
# $ conda deactivate
conda run --name opencv python -c "import sys; print(sys.version); import cv2; print(cv2.__version__)"
3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:39:48)
[GCC 9.3.0]
4.5.2
As shown in the log, the problem is with libgnutls, which must be a dependency of opencv. I don't know, and, from my reading of Anaconda's documentation is not possible to tell really, but that's a separate issue. Also, you install everything from conda-forge, whereas a more typical case is to install from both default and conda-forge. In my case, I'm also using anaconda channel, because pytest available from conda-forge is not installable with the rest of the requirements.
Anyways, the output of conda info and conda list can be found here: https://github.com/drcandacemakedamoore/cleanX/runs/3155242339?check_suite_focus=true Look for steps that are called conda info and conda list.
An obvious discrepancy between what I have, and your example is that I'm not given a chance to install opencv==4.5.2. It goes for 4.5.1, which is the version that's broken.
The short answer is that we don't really support mixed channels especially for this package. To install packages from conda-forge, you should prefer conda-forge.
I would be open to "adjusting" this package so that it can be installed in a mixed channel environment, but I can't devote design time to it. I would review a PR if you wanted to make one.
While I know that I can look for things. The information you point to is that of your base environment and does not list anything related to opencv
# packages in environment at /usr/local/miniconda:
#
# Name Version Build Channel
beautifulsoup4 4.9.3 pyhb0f4dca_0 anaconda
brotlipy 0.7.0 py39h89e85a6_1001 conda-forge
bzip2 1.0.8 h1de35cc_0 anaconda
ca-certificates 2020.10.14 0 anaconda
certifi 2021.5.30 py39h6e9494a_0 conda-forge
cffi 1.14.6 py39hb71fe58_0 conda-forge
chardet 3.0.4 py39h2c36a5b_1008 conda-forge
conda 4.10.3 py39h6e9494a_0 conda-forge
conda-build 3.21.4 py39h6e9494a_0 conda-forge
conda-package-handling 1.7.3 py39h89e85a6_0 conda-forge
cryptography 3.4.7 py39ha2c9959_0 conda-forge
filelock 3.0.12 py_0 anaconda
glob2 0.7 py_0 anaconda
icu 58.2 h0a44026_3 anaconda
idna 2.10 py_0
jinja2 2.11.2 py_0 anaconda
libarchive 3.4.2 haa3ed63_0 anaconda
libcxx 10.0.0 1
libedit 3.1.20191231 h1de35cc_1
libffi 3.3 hb1e8313_2
libiconv 1.16 h1de35cc_0 anaconda
liblief 0.10.1 h23ab428_1
libxml2 2.9.10 h7cdb67c_3 anaconda
lz4-c 1.9.2 h79c402e_3 anaconda
markupsafe 2.0.1 py39h89e85a6_0 conda-forge
ncurses 6.2 h0a44026_1
openssl 1.1.1k h0d85af4_0 conda-forge
pip 21.2.1 pyhd8ed1ab_0 conda-forge
pkginfo 1.7.1 pyhd8ed1ab_0 conda-forge
psutil 5.8.0 py39h89e85a6_1 conda-forge
py-lief 0.10.1 py39h23ab428_1
pycosat 0.6.3 py39h89e85a6_1006 conda-forge
pycparser 2.20 py_2
pyopenssl 19.1.0 pyhd3eb1b0_1
pysocks 1.7.1 py39h6e9494a_3 conda-forge
python 3.9.5 h88f2d9e_3
python-libarchive-c 2.9 py_0 anaconda
python.app 1.3 py39h89e85a6_5 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytz 2020.1 py_0 anaconda
pyyaml 5.4.1 py39hcbf5805_0 conda-forge
readline 8.0 h1de35cc_0
requests 2.24.0 py_0
ripgrep 12.1.1 0 anaconda
ruamel_yaml 0.15.80 py39h89e85a6_1004 conda-forge
setuptools 49.6.0 py39h6e9494a_3 conda-forge
six 1.15.0 py_0 anaconda
soupsieve 2.0.1 py_0 anaconda
sqlite 3.36.0 hce871da_0
tk 8.6.10 hb0a8c7a_0
tqdm 4.51.0 pyhd3eb1b0_0
tzdata 2020b h7b6447c_0 anaconda
urllib3 1.25.11 py_0
wheel 0.35.1 pyhd3eb1b0_0
xz 5.2.5 h1de35cc_0
yaml 0.2.5 haf1e3a3_0
zlib 1.2.11 h1de35cc_3
zstd 1.4.4 h1990bb4_3 anaconda
can you please help us help you by simply copy pasting the information here? You may cross reference for completeness, but I do think that it is a small ask that really helps us out.
The information you point to is that of your base environment and does not list anything related to opencv
Yes. My bad. I forgot the -n. I'll add it and rerun the job.
Well, you see the need for "mixed" environment (it's not just some Joe Nobody's channel, it's Anaconda's official channel!) is very real. Like I said, pytest isn't installable from conda-forge. Why it is like that I have no idea, and the tools are really bad to allow to analyze this in the time frame that I have. So, your typical user will end up having to mix channels.
So, should I instead file a bug with pytest on conda-forge?
Secondly, this really means that while 4.5.2 version of opencv maybe isn't broken (while not really installable for the majority of the userbase), the one that is installable, is broken because of how it defines dependencies on TLS library.
I don't know the library well enough to tell where exactly and what exactly you need to change to prevent these two libraries from being installed together, but this is definitely doable and needs to be done.
OK, so now pytests installs fine in CI, but not on my computer, with basically the same version of everything... I'll wait for it to break again, I guess.
I'm not sure what the specific problem you are hitting on your personal environment. the information that we request in the Issue template typically gets me 90% of the way to identifying the issue.
I personally install opencv and pytest everyday. Maybe you have a specific combination of versions that you've pinned to. I'm just guessing at this point.
Either way, specifying the details really helps.
please copy paste them here.
The reason I decided to use pytest from Anaconda is that it didn't work when using conda-forge...
Essentially, my experience was more or less along these lines: when using Python 3.7, "solving environment" takes about an hour, and then if you leave it to brew for about a day, you get several screenfuls of meaningless messages, where conda is just listing the packages it knows. I'm sorry, but it really takes more than a day to get it to produce the final error. I'm running this right now, but I'll have to go to sleep soon. Hopefully, there will be some results tomorrow.
I think, the problem may be created by me working on the same package and installing the same dependencies over and over in the same environment. But, this is just a guess.
this is a fair criticism of conda-forge. i think the only real tip to is to use mamba. conda-forge has really exploded the scope of the conda solver. mamba really helps.
Well, what can I say? It worked this time. It took about 6 hours to solve the environment, but, in the end, the installation succeeded. So, I have nothing to report.
as you wish.
in case you are looking for it. here is the information about mamba
I'm closing this as this was a usability issue in 2021.
We since have made conda-libmamba solver the new default and this has helped with environment resolutions for users.