another fix for the mulled build memory limitations
siebrenf opened this issue · 0 comments
siebrenf commented
I might say some naive and incorrect things. Feel free to correct me or close this :)
The memory issues in the mulled build seems to be caused by two factors:
- the solver's memory usage
- the package installation memory usage
The recent PRs to fix problem 1 (galaxy/14770 and #810) seem like a great step, but mamba would eventually run into the same issues as repositories grow. Furthermore, they do not address problem 2.
To solve problem 1, why not solve the environment outside of the mulled build container and then pass the solved environment to the container? If there are bigger containers available, problem 2 can be solved by estimating installation memory usage beforehand and assign an appropriate container size. Alternatively, this value could be used to skip doomed mulled builds. Assuming that the size of all dependency tarballs correlates to the memory usage during installation, we can obtain this fairly easily.
Here is a proof of principle, based on a package requiring only bioconda-utils as dependency:
# remove all downloaded packages to get the total download size
conda clean -afyq
# solve the environment
mamba create -n dryrun -d bioconda-utils > dryrun.txt
# obtain the solved_enironment.yaml
echo "dependencies:" > solved_enironment.yaml
grep + dryrun.txt | \
awk -F ' +' '{print " - " $6 "::" $3 "==" $4 "=" $5;}' \
>> solved_enironment.yaml
# obtain the download size
grep "Total download" dryrun.txt | \
awk -F ': ' '{print $2;}' > size.txt
# clean up
rm dryrun.txt
The results
size.txt:
249MB
solved_enironment.yaml:
dependencies:
- conda-forge/linux-64::_libgcc_mutex==0.1=conda_forge
- conda-forge/linux-64::_openmp_mutex==4.5=2_gnu
- conda-forge/noarch::aiofiles==0.8.0=pyhd8ed1ab_0
- conda-forge/noarch::aioftp==0.12.0=py_0
- conda-forge/linux-64::aiohttp==3.8.3=py37h540881e_0
- conda-forge/noarch::aiohttp-jinja2==1.1.2=py_0
- conda-forge/noarch::aiohttp-security==0.4.0=py_0
- conda-forge/noarch::aiohttp-session==2.7.0=py_0
- conda-forge/noarch::aiosignal==1.2.0=pyhd8ed1ab_0
- conda-forge/noarch::alabaster==0.7.12=py_0
- conda-forge/noarch::amqp==5.1.1=pyhd8ed1ab_0
- conda-forge/noarch::anaconda-client==1.6.14=py_0
- conda-forge/noarch::anyio==3.6.1=pyhd8ed1ab_1
- conda-forge/noarch::argh==0.26.2=pyh9f0ad1d_1002
- conda-forge/noarch::async-timeout==4.0.2=pyhd8ed1ab_0
- conda-forge/noarch::asynctest==0.13.0=py_0
- conda-forge/linux-64::atk-1.0==2.36.0=h3371d22_4
- conda-forge/noarch::attrs==22.1.0=pyh71513ae_1
- conda-forge/noarch::babel==2.10.3=pyhd8ed1ab_0
- conda-forge/noarch::backoff==1.6.0=py_1
- conda-forge/noarch::backports==1.0=py_2
- conda-forge/noarch::backports.functools_lru_cache==1.6.4=pyhd8ed1ab_0
- conda-forge/linux-64::beautifulsoup4==4.6.3=py37_1000
- conda-forge/linux-64::billiard==3.6.4.0=py37h540881e_2
- bioconda/noarch::bioconda-utils==1.1.3=pyhdfd78af_0
- conda-forge/noarch::boa==0.9.0=pyha770c72_0
- conda-forge/noarch::boltons==18.0.1=py_0
- conda-forge/linux-64::brotlipy==0.7.0=py37h540881e_1004
- conda-forge/linux-64::bzip2==1.0.8=h7f98852_4
- conda-forge/linux-64::c-ares==1.18.1=h7f98852_0
- conda-forge/linux-64::ca-certificates==2022.9.24=ha878542_0
- conda-forge/noarch::cached-property==1.5.2=hd8ed1ab_1
- conda-forge/noarch::cached_property==1.5.2=pyha770c72_1
- conda-forge/noarch::cachetools==3.0.0=py_0
- conda-forge/linux-64::cairo==1.16.0=ha61ee94_1014
- conda-forge/noarch::celery==5.2.7=pyhd8ed1ab_0
- conda-forge/noarch::certifi==2022.9.24=pyhd8ed1ab_0
- conda-forge/linux-64::cffi==1.15.1=py37h43b0acd_0
- conda-forge/linux-64::chardet==3.0.4=py37he5f6b98_1008
- conda-forge/noarch::charset-normalizer==2.1.1=pyhd8ed1ab_0
- conda-forge/linux-64::click==8.1.3=py37h89c1867_0
- conda-forge/noarch::click-didyoumean==0.3.0=pyhd8ed1ab_0
- conda-forge/noarch::click-plugins==1.1.1=py_0
- conda-forge/noarch::click-repl==0.2.0=pyhd8ed1ab_0
- conda-forge/noarch::clyent==1.2.2=py_1
- conda-forge/noarch::colorama==0.4.5=pyhd8ed1ab_0
- conda-forge/linux-64::colorlog==3.1.4=py37_1000
- conda-forge/noarch::commonmark==0.9.1=py_0
- conda-forge/linux-64::conda==4.12.0=py37h89c1867_0
- conda-forge/linux-64::conda-build==3.21.8=py37h89c1867_1
- conda-forge/noarch::conda-forge-pinning==2022.08.25.15.20.42=hd8ed1ab_0
- conda-forge/linux-64::conda-package-handling==1.9.0=py37h540881e_0
- conda-forge/linux-64::conda-verify==3.1.1=py37h89c1867_1004
- conda-forge/linux-64::cryptography==38.0.2=py37h38fbfac_0
- conda-forge/linux-64::curl==7.85.0=h7bff187_0
- conda-forge/noarch::dataclasses==0.8=pyhc8e2a94_3
- conda-forge/noarch::decorator==4.4.2=py_0
- conda-forge/linux-64::docutils==0.19=py37h89c1867_0
- conda-forge/linux-64::expat==2.4.9=h27087fc_0
- conda-forge/noarch::filelock==3.8.0=pyhd8ed1ab_0
- conda-forge/noarch::font-ttf-dejavu-sans-mono==2.37=hab24e00_0
- conda-forge/noarch::font-ttf-inconsolata==3.000=h77eed37_0
- conda-forge/noarch::font-ttf-source-code-pro==2.038=h77eed37_0
- conda-forge/noarch::font-ttf-ubuntu==0.83=hab24e00_0
- conda-forge/linux-64::fontconfig==2.14.0=hc2a2eb6_1
- conda-forge/noarch::fonts-conda-ecosystem==1=0
- conda-forge/noarch::fonts-conda-forge==1=0
- conda-forge/linux-64::freetype==2.12.1=hca18f0e_0
- conda-forge/linux-64::fribidi==1.0.10=h36c2ea0_0
- conda-forge/linux-64::frozenlist==1.3.1=py37h540881e_0
- conda-forge/linux-64::future==0.18.2=py37h89c1867_5
- bioconda/noarch::galaxy-lib==19.5.2=pyh864c0ab_1
- conda-forge/linux-64::gdk-pixbuf==2.42.8=hff1cb4f_1
- conda-forge/linux-64::gettext==0.19.8.1=h27087fc_1009
- conda-forge/noarch::gidgethub==3.0.0=py_0
- conda-forge/linux-64::giflib==5.2.1=h36c2ea0_2
- conda-forge/linux-64::git==2.38.0=pl5321h5fbbf19_0
- conda-forge/noarch::gitdb2==3.0.2=py_0
- conda-forge/noarch::gitpython==3.0.8=py_0
- conda-forge/noarch::glob2==0.7=py_0
- conda-forge/linux-64::graphite2==1.3.13=h58526e2_1001
- conda-forge/linux-64::graphviz==6.0.1=h5abf519_0
- conda-forge/linux-64::gtk2==2.24.33=h90689f9_2
- conda-forge/linux-64::gts==0.7.6=h64030ff_2
- conda-forge/linux-64::harfbuzz==5.3.0=h418a68e_0
- conda-forge/linux-64::icu==70.1=h27087fc_0
- conda-forge/linux-64::idna==2.8=py37_1000
- conda-forge/noarch::imagesize==1.4.1=pyhd8ed1ab_0
- conda-forge/linux-64::importlib-metadata==4.11.4=py37h89c1867_0
- bioconda/linux-64::involucro==1.1.2=he881be0_3
- conda-forge/noarch::jinja2==2.11.3=pyhd8ed1ab_2
- conda-forge/noarch::joblib==1.2.0=pyhd8ed1ab_0
- conda-forge/linux-64::jpeg==9e=h166bdaf_2
- conda-forge/noarch::json5==0.9.5=pyh9f0ad1d_0
- conda-forge/linux-64::jsonschema==2.6.0=py37_1002
- conda-forge/linux-64::jupyter_core==4.11.1=py37h89c1867_0
- conda-forge/linux-64::keyutils==1.6.1=h166bdaf_0
- conda-forge/linux-64::kombu==5.2.4=py37h89c1867_1
- conda-forge/linux-64::krb5==1.19.3=h3790be6_0
- conda-forge/linux-64::ld_impl_linux-64==2.36.1=hea4e1c9_2
- conda-forge/linux-64::lerc==4.0.0=h27087fc_0
- conda-forge/linux-64::libarchive==3.5.2=hb890918_3
- conda-forge/linux-64::libblas==3.9.0=16_linux64_openblas
- conda-forge/linux-64::libcblas==3.9.0=16_linux64_openblas
- conda-forge/linux-64::libcurl==7.85.0=h7bff187_0
- conda-forge/linux-64::libdeflate==1.14=h166bdaf_0
- conda-forge/linux-64::libedit==3.1.20191231=he28a2e2_2
- conda-forge/linux-64::libev==4.33=h516909a_1
- conda-forge/linux-64::libffi==3.4.2=h7f98852_5
- conda-forge/linux-64::libgcc-ng==12.1.0=h8d9b700_16
- conda-forge/linux-64::libgd==2.3.3=h18fbbfe_3
- conda-forge/linux-64::libgfortran-ng==12.1.0=h69a702a_16
- conda-forge/linux-64::libgfortran5==12.1.0=hdcd56e2_16
- conda-forge/linux-64::libglib==2.74.0=h7a41b64_0
- conda-forge/linux-64::libgomp==12.1.0=h8d9b700_16
- conda-forge/linux-64::libiconv==1.17=h166bdaf_0
- conda-forge/linux-64::liblapack==3.9.0=16_linux64_openblas
- conda-forge/linux-64::liblief==0.12.2=h27087fc_0
- conda-forge/linux-64::libmamba==0.21.2=h3985d26_0
- conda-forge/linux-64::libmambapy==0.21.2=py37hab1676c_0
- conda-forge/linux-64::libnghttp2==1.47.0=hdcd2b5c_1
- conda-forge/linux-64::libnsl==2.0.0=h7f98852_0
- conda-forge/linux-64::libopenblas==0.3.21=pthreads_h78a6416_3
- conda-forge/linux-64::libpng==1.6.38=h753d276_0
- conda-forge/linux-64::librsvg==2.54.4=h7abd40a_0
- conda-forge/linux-64::libsolv==0.7.22=h6239696_0
- conda-forge/linux-64::libsqlite==3.39.4=h753d276_0
- conda-forge/linux-64::libssh2==1.10.0=haa6b8db_3
- conda-forge/linux-64::libstdcxx-ng==12.1.0=ha89aaad_16
- conda-forge/linux-64::libtiff==4.4.0=h55922b4_4
- conda-forge/linux-64::libtool==2.4.6=h9c3ff4c_1008
- conda-forge/linux-64::libuuid==2.32.1=h7f98852_1000
- conda-forge/linux-64::libwebp==1.2.4=h522a892_0
- conda-forge/linux-64::libwebp-base==1.2.4=h166bdaf_0
- conda-forge/linux-64::libxcb==1.13=h7f98852_1004
- conda-forge/linux-64::libxml2==2.10.2=h7463322_2
- conda-forge/linux-64::libzlib==1.2.12=h166bdaf_4
- conda-forge/linux-64::lz4-c==1.9.3=h9c3ff4c_1
- conda-forge/linux-64::lzo==2.10=h516909a_1000
- conda-forge/linux-64::mamba==0.21.2=py37h6dacc13_0
- conda-forge/noarch::markdown==3.4.1=pyhd8ed1ab_0
- conda-forge/linux-64::markupsafe==2.0.1=py37h5e8e339_1
- conda-forge/linux-64::multidict==6.0.2=py37h540881e_1
- conda-forge/noarch::nbformat==5.7.0=pyhd8ed1ab_0
- conda-forge/linux-64::ncurses==6.3=h27087fc_1
- conda-forge/noarch::networkx==2.5.1=pyhd8ed1ab_0
- conda-forge/linux-64::numpy==1.19.5=py37h3e96413_3
- conda-forge/linux-64::openssl==1.1.1q=h166bdaf_0
- conda-forge/noarch::packaging==21.3=pyhd8ed1ab_0
- conda-forge/linux-64::pandas==0.23.4=py37h637b7d7_1000
- conda-forge/linux-64::pango==1.50.11=h382ae3d_0
- conda-forge/linux-64::patchelf==0.15.0=h58526e2_0
- conda-forge/linux-64::pcre2==10.37=hc3806b6_1
- conda-forge/linux-64::perl==5.32.1=2_h7f98852_perl5
- conda-forge/noarch::pip==22.2.2=pyhd8ed1ab_0
- conda-forge/linux-64::pixman==0.40.0=h36c2ea0_0
- conda-forge/noarch::pkginfo==1.8.3=pyhd8ed1ab_0
- conda-forge/noarch::prompt-toolkit==3.0.31=pyha770c72_0
- conda-forge/noarch::prompt_toolkit==3.0.31=hd8ed1ab_0
- conda-forge/linux-64::psutil==5.9.2=py37h540881e_0
- conda-forge/linux-64::pthread-stubs==0.4=h36c2ea0_1001
- conda-forge/linux-64::py-lief==0.12.2=py37hd23a5d3_0
- conda-forge/noarch::pyaml==17.12.1=py_0
- conda-forge/noarch::pybind11-abi==4=hd8ed1ab_3
- conda-forge/linux-64::pycosat==0.6.3=py37h540881e_1010
- conda-forge/noarch::pycparser==2.21=pyhd8ed1ab_0
- conda-forge/noarch::pygments==2.13.0=pyhd8ed1ab_0
- conda-forge/noarch::pyjwt==2.5.0=pyhd8ed1ab_0
- conda-forge/noarch::pyopenssl==22.1.0=pyhd8ed1ab_0
- conda-forge/noarch::pyparsing==3.0.9=pyhd8ed1ab_0
- conda-forge/linux-64::pysocks==1.7.1=py37h89c1867_5
- conda-forge/linux-64::python==3.7.12=hb7a2778_100_cpython
- conda-forge/noarch::python-dateutil==2.8.2=pyhd8ed1ab_0
- conda-forge/noarch::python-fastjsonschema==2.16.2=pyhd8ed1ab_0
- conda-forge/linux-64::python-libarchive-c==4.0=py37h89c1867_1
- conda-forge/linux-64::python_abi==3.7=2_cp37m
- conda-forge/noarch::pytz==2022.4=pyhd8ed1ab_0
- conda-forge/linux-64::pyyaml==6.0=py37h540881e_4
- conda-forge/linux-64::readline==8.1.2=h0f457ee_0
- conda-forge/linux-64::regex==2018.08.29=py37h14c3975_1000
- conda-forge/linux-64::reproc==14.2.3=h7f98852_0
- conda-forge/linux-64::reproc-cpp==14.2.3=h9c3ff4c_0
- conda-forge/linux-64::requests==2.22.0=py37_1
- conda-forge/noarch::rich==12.6.0=pyhd8ed1ab_0
- conda-forge/linux-64::ripgrep==13.0.0=h2f28480_2
- conda-forge/linux-64::ruamel.yaml==0.17.21=py37h540881e_1
- conda-forge/linux-64::ruamel.yaml.clib==0.2.6=py37h540881e_1
- conda-forge/linux-64::ruamel_yaml==0.15.80=py37h540881e_1007
- conda-forge/linux-64::setuptools==59.8.0=py37h89c1867_1
- conda-forge/linux-64::simplejson==3.17.6=py37h540881e_1
- conda-forge/noarch::six==1.16.0=pyh6c4a22f_0
- conda-forge/linux-64::skopeo==0.1.35=haa36a5b_1
- conda-forge/noarch::smmap==5.0.0=pyhd8ed1ab_0
- conda-forge/noarch::smmap2==3.0.1=py_0
- conda-forge/noarch::sniffio==1.3.0=pyhd8ed1ab_0
- conda-forge/noarch::snowballstemmer==2.2.0=pyhd8ed1ab_0
- conda-forge/noarch::sphinx==5.1.1=pyhd8ed1ab_1
- conda-forge/noarch::sphinx-autodoc-typehints==1.19.2=pyhd8ed1ab_0
- conda-forge/noarch::sphinxcontrib-applehelp==1.0.2=py_0
- conda-forge/noarch::sphinxcontrib-devhelp==1.0.2=py_0
- conda-forge/noarch::sphinxcontrib-htmlhelp==2.0.0=pyhd8ed1ab_0
- conda-forge/noarch::sphinxcontrib-jsmath==1.0.1=py_0
- conda-forge/noarch::sphinxcontrib-qthelp==1.0.3=py_0
- conda-forge/noarch::sphinxcontrib-serializinghtml==1.1.5=pyhd8ed1ab_2
- conda-forge/linux-64::sqlite==3.39.4=h4ff8645_0
- conda-forge/linux-64::tk==8.6.12=h27826a3_0
- conda-forge/noarch::tqdm==4.64.1=pyhd8ed1ab_0
- conda-forge/noarch::traitlets==5.4.0=pyhd8ed1ab_0
- conda-forge/noarch::typing-extensions==4.4.0=hd8ed1ab_0
- conda-forge/noarch::typing_extensions==4.4.0=pyha770c72_0
- conda-forge/noarch::uritemplate==4.1.1=pyhd8ed1ab_0
- conda-forge/noarch::urllib3==1.25.11=py_0
- conda-forge/noarch::vine==5.0.0=pyhd8ed1ab_1
- conda-forge/noarch::watchgod==0.8.2=pyhd8ed1ab_0
- conda-forge/noarch::wcwidth==0.2.5=pyh9f0ad1d_2
- conda-forge/noarch::wheel==0.37.1=pyhd8ed1ab_0
- conda-forge/linux-64::xorg-kbproto==1.0.7=h7f98852_1002
- conda-forge/linux-64::xorg-libice==1.0.10=h7f98852_0
- conda-forge/linux-64::xorg-libsm==1.2.3=hd9c2040_1000
- conda-forge/linux-64::xorg-libx11==1.7.2=h7f98852_0
- conda-forge/linux-64::xorg-libxau==1.0.9=h7f98852_0
- conda-forge/linux-64::xorg-libxdmcp==1.1.3=h7f98852_0
- conda-forge/linux-64::xorg-libxext==1.3.4=h7f98852_1
- conda-forge/linux-64::xorg-libxrender==0.9.10=h7f98852_1003
- conda-forge/linux-64::xorg-renderproto==0.11.1=h7f98852_1002
- conda-forge/linux-64::xorg-xextproto==7.3.0=h7f98852_1002
- conda-forge/linux-64::xorg-xproto==7.0.31=h7f98852_1007
- conda-forge/linux-64::xz==5.2.6=h166bdaf_0
- conda-forge/linux-64::yaml==0.2.5=h7f98852_2
- conda-forge/linux-64::yaml-cpp==0.6.3=he1b5a44_4
- conda-forge/linux-64::yarl==1.7.2=py37h540881e_2
- conda-forge/noarch::zipp==3.9.0=pyhd8ed1ab_0
- conda-forge/linux-64::zlib==1.2.12=h166bdaf_4
- conda-forge/linux-64::zstd==1.5.2=h6239696_4