make package available in PyPI
Closed this issue · 14 comments
Hi,
I am running a CI job to test one of my packages that depends on mobie.
I'd like to install mobie from a dependencies list using pip.
Would it be possible to make it available on PyPI?
same holds for https://github.com/constantinpape/pybdv/
Although there it seems that PyPI provides an old version.
Unfortunately this is not currently possible for mobie-utils-python, since it has several dependencies that are not available via PyPI and that are very difficult to provide through it in a reliable fashion:
- https://github.com/constantinpape/z5
- https://github.com/DerThorsten/nifty
- https://github.com/ukoethe/vigra
These are c++ packages with python bindings, for which building wheels is not straight forward.
There are some efforts to enable building wheels from conda recipes, e.g. https://github.com/conda-incubator/conda-press.
But I don't know how well this would work in these cases and I don't have time to look into this.
same holds for https://github.com/constantinpape/pybdv/
That's possible since it does not rely on any upstream dependencies that are not available on PyPI.
Although there it seems that PyPI provides an old version.
Really? Could you link to it? I couldn't find it; and it's not by me.
I am running a CI job to test one of my packages that depends on mobie.
If you're using GH Actions then using conda is very easy: https://github.com/marketplace/actions/setup-miniconda
I am using Gitlab CI, because I need to make some custom-built containers available as web service APIs to run my tests against.
I will try to base my test run on anacoda then and get my dependencies through there.
I think I had an older version of pybdv installed in that environment, so when running pip install it did simply fetch the local installation.
In a fresh clean environment it does not find it, so false alarm...
I got my conda container up and running. I am now running into a strange error when importing mobie:
import mobie
/opt/conda/lib/python3.7/site-packages/mobie/__init__.py:1: in <module>
from .image_data import add_image, add_bdv_image
/opt/conda/lib/python3.7/site-packages/mobie/image_data.py:5: in <module>
import mobie.metadata as metadata
/opt/conda/lib/python3.7/site-packages/mobie/metadata/__init__.py:1: in <module>
from .bookmark_metadata import add_additional_bookmark, add_dataset_bookmark, add_grid_bookmark
/opt/conda/lib/python3.7/site-packages/mobie/metadata/bookmark_metadata.py:6: in <module>
from .view_metadata import get_view, get_grid_view
/opt/conda/lib/python3.7/site-packages/mobie/metadata/view_metadata.py:8: in <module>
from ..tables import check_source_annotation_table, compute_source_annotation_table
/opt/conda/lib/python3.7/site-packages/mobie/tables/__init__.py:3: in <module>
from .traces_table import compute_trace_default_table
/opt/conda/lib/python3.7/site-packages/mobie/tables/traces_table.py:11: in <module>
from ..import_data.traces import parse_traces, vals_to_coords
/opt/conda/lib/python3.7/site-packages/mobie/import_data/__init__.py:5: in <module>
from .traces import import_traces
/opt/conda/lib/python3.7/site-packages/mobie/import_data/traces.py:5: in <module>
import elf.skeleton.io as skio
/opt/conda/lib/python3.7/site-packages/elf/skeleton/__init__.py:1: in <module>
from .skeletonize import skeletonize, get_method_names
/opt/conda/lib/python3.7/site-packages/elf/skeleton/skeletonize.py:1: in <module>
from .thinning import thinning
/opt/conda/lib/python3.7/site-packages/elf/skeleton/thinning.py:3: in <module>
from skan import csr
/opt/conda/lib/python3.7/site-packages/skan/__init__.py:1: in <module>
from .csr import skeleton_to_csgraph, branch_statistics, summarize, Skeleton
/opt/conda/lib/python3.7/site-packages/skan/csr.py:91: in <module>
@numba.jit(nopython=True, cache=True, nogil=True)
/opt/conda/lib/python3.7/site-packages/numba/core/decorators.py:212: in wrapper
disp.enable_caching()
/opt/conda/lib/python3.7/site-packages/numba/core/dispatcher.py:863: in enable_caching
self._cache = FunctionCache(self.py_func)
/opt/conda/lib/python3.7/site-packages/numba/core/caching.py:613: in __init__
self._impl = self._impl_class(py_func)
/opt/conda/lib/python3.7/site-packages/numba/core/caching.py:351: in __init__
"for file %r" % (qualname, source_path))
E RuntimeError: cannot cache function '_write_pixel_graph': no locator available for file '/opt/conda/lib/python3.7/site-packages/skan/csr.py'
Any idea what could cause this?
Hmm, this looks like a numba error caused by the skan import. Have you seen this one before @jni?
@martinschorb, can you try this with a more recent python version? 3.7 is fairly old, maybe that could fix it.
If not I could update elf, so that skan only gets imported on demand.
unfortunately, one of the core packages I am using still relies on 3.7, so at the moment I am stuck using that.
Also, it runs well on 3.7 in my other environments (Mac, Linux). Just inside the continuumio/miniconda3 container, it refuses to import after installing everything.
Can you run a conda list in that container and paste the output here?
The issue arises when running python as a non-privileged user inside the container.
edit: minimal example only installing mobie_utils from conda-forge, nothing else ...
# su testuser1 -c 'conda list'
# packages in environment at /opt/conda:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_llvm conda-forge
affogato 0.3.3 py37hb3b4854_0 conda-forge
aiobotocore 2.2.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.1 py37h5e8e339_0 conda-forge
aioitertools 0.10.0 pyhd8ed1ab_0 conda-forge
aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
asynctest 0.13.0 py_0 conda-forge
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
boost 1.74.0 py37h796e4cb_5 conda-forge
boost-cpp 1.74.0 h6cacc03_7 conda-forge
botocore 1.24.21 pyhd8ed1ab_1 conda-forge
brotlipy 0.7.0 py37h27cfd23_1003
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
certifi 2021.10.8 py37h89c1867_1 conda-forge
cffi 1.15.0 py37hd667e15_1
chardet 4.0.0 py37h06a4308_1003
charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
cluster_tools 0.4.1 pyhd8ed1ab_0 conda-forge
coin-or-cbc 2.10.7 h3786ebc_0 conda-forge
coin-or-cgl 0.60.3 he7e83c3_2 conda-forge
coin-or-clp 1.17.6 h256e9bb_3 conda-forge
coin-or-osi 0.108.6 h3b589db_2 conda-forge
coin-or-utils 2.11.6 h573740c_0 conda-forge
coincbc 2.10.7 0_metapackage conda-forge
conda 4.12.0 py37h89c1867_0 conda-forge
conda-package-handling 1.8.0 py37h7f8727e_0
cryptography 36.0.0 py37h9ce1e76_0
curl 7.82.0 h7bff187_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.11.2 py37h5e8e339_1 conda-forge
dask-core 1.1.4 py37_1
docutils 0.18.1 py37h89c1867_0 conda-forge
enum34 1.1.10 py37hc8dfbb8_2 conda-forge
et_xmlfile 1.0.1 py_1001 conda-forge
expat 2.4.7 h27087fc_0 conda-forge
fftw 3.3.10 nompi_h77c792f_102 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
frozenlist 1.3.0 py37h5e8e339_0 conda-forge
fsspec 2022.2.0 pyhd8ed1ab_0 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
git 2.35.1 pl5321h36853c3_0 conda-forge
glpk 4.65 h9202a9a_1004 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
h5py 3.6.0 nompi_py37hd308b1e_100 conda-forge
hdf5 1.12.1 nompi_h2386368_104 conda-forge
icu 69.1 h9c3ff4c_0 conda-forge
idna 2.10 pyhd3eb1b0_0
ilmbase 2.5.5 h780b84a_0 conda-forge
imagecodecs-lite 2019.12.3 py37hb1e94ed_4 conda-forge
imageio 2.6.1 py37_0 conda-forge
importlib-metadata 4.11.3 py37h89c1867_0 conda-forge
importlib_resources 5.6.0 pyhd8ed1ab_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jmespath 1.0.0 pyhd8ed1ab_0 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h7f98852_0 conda-forge
jsonschema 4.4.0 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.0 py37h7cecad7_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 h7274673_9
lemon 1.3.1 hf0e68b2_5 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 13_linux64_openblas conda-forge
libcblas 3.9.0 13_linux64_openblas conda-forge
libcurl 7.82.0 h7bff187_0 conda-forge
libdeflate 1.10 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1d223b6_14 conda-forge
libgfortran-ng 11.2.0 h69a702a_14 conda-forge
libgfortran5 11.2.0 h5c6108e_14 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 13_linux64_openblas conda-forge
liblapacke 3.9.0 13_linux64_openblas conda-forge
libllvm11 11.1.0 hf817b99_3 conda-forge
libnghttp2 1.47.0 h727a467_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_14 conda-forge
libtiff 4.3.0 h542a066_3 conda-forge
libwebp-base 1.2.2 h7f98852_1 conda-forge
libzlib 1.2.11 h166bdaf_1014 conda-forge
llvm-openmp 13.0.1 he0ac6c6_1 conda-forge
llvmlite 0.38.0 py37h9d7f4d0_0 conda-forge
lockfile 0.12.2 py_1 conda-forge
luigi 3.0.3 py37h06a4308_0
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
matplotlib-base 3.4.3 py37h1058ff1_0 conda-forge
mobie_utils 0.2.3 pyhd8ed1ab_0 conda-forge
mrcfile 1.3.0 pyh44b312d_0 conda-forge
multidict 6.0.2 py37h5e8e339_0 conda-forge
ncurses 6.2 he6710b0_1
networkx 2.7.1 pyhd8ed1ab_0 conda-forge
nifty 1.2.1 py37h023b2db_3 conda-forge
numba 0.55.1 py37h2d894fd_0 conda-forge
numpy 1.21.5 py37hf2998dd_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openexr 2.5.5 hf817b99_0 conda-forge
openpyxl 3.0.9 pyhd8ed1ab_0 conda-forge
openssl 1.1.1n h166bdaf_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py37he8f5f7f_0 conda-forge
pathlib 1.0.1 py37h89c1867_5 conda-forge
pcre2 10.37 h032f7d1_0 conda-forge
perl 5.32.1 2_h7f98852_perl5 conda-forge
pillow 7.2.0 py37h718be6c_2 conda-forge
pip 21.2.2 py37h06a4308_0
pybdv 0.5.3 pyhd8ed1ab_0 conda-forge
pycosat 0.6.3 py37h27cfd23_0
pycparser 2.20 py_2
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 3.0.7 pyhd8ed1ab_0 conda-forge
pyrsistent 0.18.1 py37h5e8e339_0 conda-forge
pysocks 1.7.1 py37_1
python 3.7.11 h12debd9_0
python-daemon 2.3.0 pyhd8ed1ab_1 conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-elf 0.4.1 pyhd8ed1ab_0 conda-forge
python_abi 3.7 2_cp37m conda-forge
pytz 2022.1 pyhd8ed1ab_0 conda-forge
pywavelets 1.3.0 py37h6c7ee08_0 conda-forge
readline 8.1 h27cfd23_0
requests 2.25.1 pyhd3eb1b0_0
ruamel_yaml 0.15.100 py37h27cfd23_0
s3fs 0.5.1 py_0 conda-forge
scikit-image 0.19.2 py37he8f5f7f_0 conda-forge
scikit-learn 1.0.2 py37hf9e9bfc_0 conda-forge
scipy 1.7.3 py37hf2a6cf1_0 conda-forge
setuptools 58.0.4 py37h06a4308_0
six 1.16.0 pyhd3eb1b0_0
skan 0.10.0 pyhd8ed1ab_0 conda-forge
sqlite 3.36.0 hc218d9a_0
tenacity 6.3.1 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tifffile 2019.7.26.2 py37_0 conda-forge
tk 8.6.10 hbc83047_0
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py37h5e8e339_2 conda-forge
tqdm 4.61.2 pyhd3eb1b0_1
typing-extensions 4.1.1 hd8ed1ab_0 conda-forge
typing_extensions 4.1.1 pyha770c72_0 conda-forge
tzdata 2021a h52ac0ba_0
urllib3 1.26.6 pyhd3eb1b0_1
vigra 1.11.1 py37h71a3c6e_1029 conda-forge
wheel 0.36.2 pyhd3eb1b0_0
wrapt 1.14.0 py37h540881e_0 conda-forge
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
yarl 1.7.2 py37h5e8e339_1 conda-forge
z5py 2.0.15 py37h2f61680_0 conda-forge
zipp 3.7.0 pyhd8ed1ab_1 conda-forge
zlib 1.2.11 h166bdaf_1014 conda-forge
zstd 1.5.2 ha95c52a_0 conda-forge
The issue arises when running python as a non-privileged user inside the container.
By "the issue", do you mean the error reported in #49 (comment)?
If so, then this is probably caused by numba modifying files that are not accessible in the container.
This is an issue I can't do much about; except for guarding against import errors of skan / numba in elf.
But numba is a very commonly used package, so a CI set-up that can't deal with it is not very good.
Yes, it looks like it is the good old conda permissions issue in a multi-user environment.
If I chmod 777 all conda and install everything as the user, it works.
Cool, all the test container runs now.
Any idea why I get an old mobie_utils version when installing it from conda-forge?
mobie.add_bdv_image(xmlpath,
outpath,
dataset_name=dataset_name,
image_name=image_name,
menu_name=menu_name,
> move_data=True)
E TypeError: add_bdv_image() got an unexpected keyword argument 'move_data'
Do I still need to additionally pip install from the repo?
I haven't done a release yet since you've added the move_data option. I have done it now, but it will take probably till tomorrow to be on conda-forge. I will ping you then.
Cool, thanks.