dask/hdfs3

Can not find the shared library: libhdfs3.so

iniyanp opened this issue · 5 comments

iniyanp commented

Hi,

I am trying to install hdfs3 using pip in centos. But when I try to use HDFileSystem(), it is throwing Can not find the shared library: libhdfs3.so ERROR.

Steps I followed:

curl -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo \
    && yum makecache \
    && yum install -y libhdfs3 libhdfs3-devel \
    && pip install hdfs3

I have tried installing via conda. It works. But unfortunately we can't use conda in our environment for some reasons.

Could anyone help me? Thanks

gravitywp commented

check lbhdfs3.so in LD_LIBRARY_PATH.
or libdhfs3.so version doesn't match hdfs3 needed.
or you could just compile from source https://github.com/ContinuumIO/libhdfs3-downstream

martindurant commented

To diagnose further, you could try running the ctypes load command in the lib module yourself, to see what the exact error was.

martindurant commented

Note that lib/hdfs3 has never been tested via yum by me. Since I see "wangzw" in the URL, I suspect it may be an old build. I recommend using conda for the installation of complicated python-related binary libraries.

  • 👍1
chanansh commented

I also have this problem in a mac. fresh install using conda did not help

martindurant commented

Same suggestions as above...
Note that lib/hdfs3 has not changed or been released in some time, but other libraries have moved on.