.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/libpython2.7.so: undefined symbol: PyType_GetFlags
Closed this issue · 11 comments
Hello!
Thank you for this package! Could you help me, please, to fix this with new versions of Ubuntu 22.04?
R -e "library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)"
#44 0.450 R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
#44 0.450 Copyright (C) 2023 The R Foundation for Statistical Computing
#44 0.450 Platform: x86_64-pc-linux-gnu (64-bit)
...
#44 0.537 > library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)
#44 1.652 trying URL 'https://repo.anaconda.com/miniconda/Miniconda3-py310_23.5.0-3-Linux-x86_64.sh'
#44 2.086 Content type 'application/x-sh' length 95815836 bytes (91.4 MB)
#44 2.189 ==================================================
#44 6.373 downloaded 91.4 MB
#44 6.373
#44 6.387 PREFIX=/home/rstudio/.cache/R/basilisk/1.13.3/0
#44 6.627 Unpacking payload ...
...
#44 178.5 added / updated specs:
#44 178.5 - python=2.7
#44 178.5
#44 178.5
#44 178.5 The following packages will be downloaded:
#44 178.5
#44 178.5 package | build
#44 178.5 ---------------------------|-----------------
#44 178.5 _libgcc_mutex-0.1 | conda_forge 3 KB conda-forge
#44 178.5 _openmp_mutex-4.5 | 2_gnu 23 KB conda-forge
#44 178.5 ca-certificates-2023.7.22 | hbcca054_0 146 KB conda-forge
#44 178.5 certifi-2019.11.28 | py27h8c360ce_1 149 KB conda-forge
#44 178.5 ld_impl_linux-64-2.40 | h41732ed_0 688 KB conda-forge
#44 178.5 libffi-3.2.1 | he1b5a44_1007 47 KB conda-forge
#44 178.5 libgcc-ng-13.1.0 | he5830b7_0 758 KB conda-forge
#44 178.5 libgomp-13.1.0 | he5830b7_0 409 KB conda-forge
#44 178.5 libsqlite-3.42.0 | h2797004_0 809 KB conda-forge
#44 178.5 libstdcxx-ng-13.1.0 | hfd8a6a1_0 3.7 MB conda-forge
#44 178.5 libzlib-1.2.13 | hd590300_5 60 KB conda-forge
#44 178.5 ncurses-6.4 | hcb278e6_0 860 KB conda-forge
#44 178.5 openssl-1.1.1v | hd590300_0 1.9 MB conda-forge
#44 178.5 pip-20.1.1 | pyh9f0ad1d_0 1.1 MB conda-forge
#44 178.5 python-2.7.15 |h5a48372_1011_cpython 12.2 MB conda-forge
#44 178.5 python_abi-2.7 | 1_cp27mu 4 KB conda-forge
#44 178.5 readline-8.2 | h8228510_1 275 KB conda-forge
#44 178.5 setuptools-44.0.0 | py27_0 663 KB conda-forge
#44 178.5 sqlite-3.42.0 | h2c6b66d_0 799 KB conda-forge
#44 178.5 tk-8.6.12 | h27826a3_0 3.3 MB conda-forge
#44 178.5 wheel-0.37.1 | pyhd8ed1ab_0 31 KB conda-forge
#44 178.5 zlib-1.2.13 | hd590300_5 91 KB conda-forge
#44 178.5 ------------------------------------------------------------
#44 178.5 Total: 27.8 MB
#44 178.5
#44 178.5 The following NEW packages will be INSTALLED:
#44 178.5
#44 178.5 _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
#44 178.5 _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
#44 178.5 ca-certificates conda-forge/linux-64::ca-certificates-2023.7.22-hbcca054_0
#44 178.5 certifi conda-forge/linux-64::certifi-2019.11.28-py27h8c360ce_1
#44 178.5 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0
#44 178.5 libffi conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
#44 178.5 libgcc-ng conda-forge/linux-64::libgcc-ng-13.1.0-he5830b7_0
#44 178.5 libgomp conda-forge/linux-64::libgomp-13.1.0-he5830b7_0
#44 178.5 libsqlite conda-forge/linux-64::libsqlite-3.42.0-h2797004_0
#44 178.5 libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-13.1.0-hfd8a6a1_0
#44 178.5 libzlib conda-forge/linux-64::libzlib-1.2.13-hd590300_5
#44 178.5 ncurses conda-forge/linux-64::ncurses-6.4-hcb278e6_0
#44 178.5 openssl conda-forge/linux-64::openssl-1.1.1v-hd590300_0
#44 178.5 pip conda-forge/noarch::pip-20.1.1-pyh9f0ad1d_0
#44 178.5 python conda-forge/linux-64::python-2.7.15-h5a48372_1011_cpython
#44 178.5 python_abi conda-forge/linux-64::python_abi-2.7-1_cp27mu
#44 178.5 readline conda-forge/linux-64::readline-8.2-h8228510_1
#44 178.5 setuptools conda-forge/linux-64::setuptools-44.0.0-py27_0
#44 178.5 sqlite conda-forge/linux-64::sqlite-3.42.0-h2c6b66d_0
#44 178.5 tk conda-forge/linux-64::tk-8.6.12-h27826a3_0
#44 178.5 wheel conda-forge/noarch::wheel-0.37.1-pyhd8ed1ab_0
#44 178.5 zlib conda-forge/linux-64::zlib-1.2.13-hd590300_5
...
#44 787.8 Processing /home/rstudio/R/library/crisprScore/python/azimuth
#44 787.9 Requirement already satisfied: scipy<=1.2.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.2.1)
#44 787.9 Requirement already satisfied: numpy<=1.14 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.11.3)
#44 787.9 Requirement already satisfied: matplotlib in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (2.2.3)
#44 788.0 Requirement already satisfied: nose in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.3.7)
#44 788.0 Requirement already satisfied: scikit-learn<0.18,>=0.17.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (0.17.1)
#44 788.0 Requirement already satisfied: pandas in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (0.23.4)
#44 788.0 Requirement already satisfied: biopython<=1.76 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.74)
#44 788.0 Requirement already satisfied: cycler>=0.10 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (0.10.0)
#44 788.0 Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (2.4.7)
#44 788.0 Requirement already satisfied: python-dateutil>=2.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (2.8.1)
#44 788.0 Requirement already satisfied: pytz in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (2020.1)
#44 788.0 Requirement already satisfied: six>=1.10 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (1.15.0)
#44 788.0 Requirement already satisfied: kiwisolver>=1.0.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (1.1.0)
#44 788.0 Requirement already satisfied: backports.functools_lru_cache in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (1.6.1)
#44 788.0 Requirement already satisfied: subprocess32 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (3.5.4)
#44 788.0 Requirement already satisfied: setuptools in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from kiwisolver>=1.0.1->matplotlib->Azimuth==2.0) (44.0.0.post20200102)
#44 788.0 Building wheels for collected packages: Azimuth
#44 788.0 Building wheel for Azimuth (setup.py): started
#44 788.2 Building wheel for Azimuth (setup.py): finished with status 'done'
#44 788.2 Created wheel for Azimuth: filename=Azimuth-2.0-py2-none-any.whl size=163884 sha256=cb56e2d7a8bad3018036c51eaa65e25109f5668f4c195acd3477d5618d491688
#44 788.2 Stored in directory: /tmp/pip-ephem-wheel-cache-KOxihB/wheels/56/4b/66/e075652909d01dd3d766e57ba0d9a334f4ddbcce926d80fe70
#44 788.2 Successfully built Azimuth
#44 788.2 Installing collected packages: Azimuth
#44 788.3 Successfully installed Azimuth-2.0
#44 789.3 Error: PyType_GetFlags - /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/libpython2.7.so: undefined symbol: PyType_GetFlags
#44 789.3 In addition: Warning message:
#44 789.3 In py_initialize(config$python, config$libpython, config$pythonhome, :
#44 789.3 Python 2 reached EOL on January 1, 2020. Python 2 compatability will be removed in an upcoming reticulate release.
#44 789.3 Execution halted
#44 ERROR: process "/bin/sh -c cd \"$SOFT/crisprScore\" && R -e \"library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)\"" did not complete successfully: exit code: 1
Maybe this is a reason: https://foss.heptapod.net/pypy/pypy/-/issues/3173#note_160826 ?
I suppose, that some versions should be updated maybe here:
https://github.com/crisprVerse/crisprScore/blob/master/R/basilisk.R#L1
Is this the most recent crisprScore (1.4 release, 1.5 devel)?
I thought that the latest version was no longer using reticulate to load Python2 environments.
I have tried both the release and the latest master
Oh wait, I see you're actually calling basiliskStart(env_azimuth). You shouldn't be doing that; from the output, you can see that reticulate's support for Python 2 is on its last legs. Instead, getAzimuthScores just uses the Conda environment's Python directly.
Incidentally, @Jfortin1 it may be safer to be more explicit about the Python used in
crisprScore/R/getAzimuthScores.R
Line 86 in 0bef9c4
e.g., with the full path via basilisk.utils:::getPythonBinary(env).
According to this repo, the crisprScore's release is 1.1.17. Master branch is newer, that's why I have tested it too.
With basiliskStart(env_azimuth) I simply try to install all needed environment. I am creating a docker image and want all installation to be done during build step for not to wait for the environment installation every run.
When I ran the same commands in the beginning of 2023 year, everything worked.
I suppose the problem is in https://github.com/crisprVerse/crisprScore/blob/master/R/basilisk.R#L184 or https://github.com/crisprVerse/crisprScore/blob/master/R/basilisk.R#L1
To install up-front, you should set the BASILISK_USE_SYSTEM_DIR environment variable, as documented here.
Behavior at the start of the year doesn't really mean much, given reticulate has updated a few times since then (possibly dropping Python 2 support along the way) and conda itself seems to rebuild its binaries every now and then.
FIxed now in v1.5.1
If I use BASILISK_USE_SYSTEM_DIR=1, problem looks different:
> library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)
#44 1.672 Error in obtainEnvironmentPath(env) :
#44 1.672 environment 'azimuth_basilisk' should have been created during 'crisprScore' installation
#44 1.672 Calls: basiliskStart -> obtainEnvironmentPath
#44 1.672 Execution halted> library(basilisk); source('R/basilisk.R'); obtainEnvironmentPath(env_azimuth)
Error in obtainEnvironmentPath(env_azimuth) :
environment 'azimuth_basilisk' should have been created during 'crisprScore' installation
Execution haltedI don't know how you're setting up your Dockerfile but you need to set this variable before installation of basilisk and crisprScore.
Thank you! Now I understood, that I should set ENV VAR BASILISK_USE_SYSTEM_DIR=1 before installing basilisk:
R -e "remotes::install_github('LTLA/basilisk', ref='master'); remotes::install_github('LTLA/basilisk.utils', ref='master')"!
After this I should install crisprVerse:
R -e "remotes::install_github('crisprVerse/crisprBase', ref='master'); remotes::install_github('crisprVerse/crisprBowtie', ref='master'); remotes::install_github('crisprVerse/crisprScoreData', ref='master'); remotes::install_github('crisprVerse/crisprBwa', ref='master'); remotes::install_github('crisprVerse/crisprDesign', ref='master'); remotes::install_github('crisprVerse/crisprDesignData', ref='master'); remotes::install_github('crisprVerse/crisprViz', ref='master'); remotes::install_github('crisprVerse/crisprVerse', ref='main'); remotes::install_github('crisprVerse/crisprScore', ref='master')".
And finally I should NOT run cd "$SOFT/crisprScore" && R -e "library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)", it is not necessary (and fails with error).
Thank you! It works!