faiss fails to import on python==3.10.15
Closed this issue · 18 comments
Summary
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/__init__.py", line 16, in <module>
from .loader import *
File "/data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/loader.py", line 134, in <module>
from .swigfaiss import *
File "/data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/swigfaiss.py", line 13, in <module>
from . import _swigfaiss
ImportError: /data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/../../../libfaiss.so: undefined symbol: __libc_single_threaded
Platform
OS: Ubuntu 20.04.6 LTS
Faiss version: 1.9.0
Installed from: anaconda
Faiss compilation options:
Running on:
- CPU
- GPU
Interface:
- C++
- Python
Reproduction instructions
import faiss
me too
I install faiss using: conda install pytorch::faiss-gpu and encounter the same error
Hi, for me to sanity check, can you paste the steps you used to install?
Following https://github.com/facebookresearch/faiss/wiki/Installing-Faiss, I didn't see the issue, but my conda env is already set up.
I tried also:
conda install -c pytorch faiss-cpu=1.9.0
python
>>> import faiss
without errors
@mnorris11
Thank you so much for answering my question!
I followed your installation method and was able to import the "faiss" package successfully. However, when I run the program, I get an error that there is no "StandardGpuResources" module.
Subsequently, I was using the "conda install pytorch::faiss-gpu" command to install the faiss-gpu version, and when I imported the faiss package, I had the following problem:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/__init__.py", line 16, in <module>
from .loader import *
File "/data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/loader.py", line 134, in <module>
from .swigfaiss import *
File "/data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/swigfaiss.py", line 13, in <module>
from . import _swigfaiss
ImportError: /data1/users/yangcong/miniconda3/envs/ccvit/lib/python3.10/site-packages/faiss/../../../libfaiss.so: undefined symbol: __libc_single_threaded
I tried these steps and could not reproduce with a fresh conda env:
$ conda create -n faiss_3943
$ conda update -y -n base -c defaults conda
$ conda activate faiss_3943
$ conda install -y -q python=3.11 cmake make swig mkl=2023 mkl-devel=2023 numpy scipy pytest gxx_linux-64=11.2 sysroot_linux-64 gflags
$ conda install pytorch::faiss-gpu
$ python
Python 3.11.10 (main, Oct 3 2024, 07:29:13) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
>>>
This is on CentOS. This does not work for you?
@mnorris11
Thanks, I followed your steps, but I still meet the following problems.
Driver Version: 525.147.05 CUDA Version: 12.0
>>> import faiss
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/chimao/anaconda3/envs/fgpu/lib/python3.11/site-packages/faiss/__init__.py", line 16, in <module>
from .loader import *
File "/home/chimao/anaconda3/envs/fgpu/lib/python3.11/site-packages/faiss/loader.py", line 134, in <module>
from .swigfaiss import *
File "/home/chimao/anaconda3/envs/fgpu/lib/python3.11/site-packages/faiss/swigfaiss.py", line 13, in <module>
from . import _swigfaiss
ImportError: /home/chimao/anaconda3/envs/fgpu/lib/python3.11/site-packages/faiss/../../../libfaiss.so: undefined symbol: __libc_single_threaded
Can you paste your conda list -n fgpu and ldd --version? We can check the some versions.
FYI there is also this ongoing issue: #3925 (comment)
Can you paste your
conda list -n fgpuandldd --version? We can check the some versions.
the ldd --version is 2.17.
@mnorris11 Thank you so much for your reply! I deleted the previous environment and then successfully installed version 1.8.0 of faiss-gpu by following the steps below:
conda create --name faiss_1.8.0
conda activate faiss_1.8.0
conda install -c pytorch -c nvidia faiss-gpu=1.8.0 pytorch=*=*cuda* pytorch-cuda=11 numpy
Cool, glad to hear it's installed. I will close the issue then.
We saw from other issues online that if you see this error about __libc_single_threaded, glibc may need to be upgraded on your system.
Hi , I am running into the same issue when I tried to import faiss
>>> import faiss
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/projectnb/dnn-motion/yiweng/conda/envs/mjepa/lib/python3.10/site-packages/faiss/__init__.py", line 16, in <module>
from .loader import *
File "/projectnb/dnn-motion/yiweng/conda/envs/mjepa/lib/python3.10/site-packages/faiss/loader.py", line 134, in <module>
from .swigfaiss import *
File "/projectnb/dnn-motion/yiweng/conda/envs/mjepa/lib/python3.10/site-packages/faiss/swigfaiss.py", line 13, in <module>
from . import _swigfaiss
ImportError: /projectnb/dnn-motion/yiweng/conda/envs/mjepa/lib/python3.10/site-packages/faiss/../../../libfaiss.so: undefined symbol: __libc_single_threaded
I installed faiss-gpu via conda as instructed and my version is 1.9.0
My ldd --version is 2.28
I don't know if this is relevant but my pre-existing conda environment has torch installed via pip. ---- hope that I don't need to reinstall everything?
>>> python -c 'import torch; print(torch.__version__)'
2.5.1+cu124
My OS is AlmaLinux 8
Any advice will be appreciated! Thank you!!
Hi @EvenGu , if you run conda list, what is the gxx version?
On my system right now, it looks like libgcc 14.2.0
Hi, @mnorris11,
Sorry for my late reply.
Checking my libgcc in conda, I got
>>>conda list libgcc
_libgcc_mutex 0.1
libgcc-ng 11.2.0
I actually do not have a libgcc package... So I proceed with the installation and now it returns:
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
However, the error stays the same when I import faiss...
@EvenGu did you create a fresh conda environment following the steps above, still didn't work? Same __libc_single_threaded error?
Hi @mnorris11,
I am happy to report that with
conda install -c pytorch -c nvidia faiss-gpu=1.8.0 pytorch="*=*cuda*" pytorch-cuda=12 numpy
everything works.
However, if I try the version 1.9.0, it does not.
Probably something changed in the version update that caused that import error.
Same here. I used Miniconda. And after trying:
conda install -c pytorch -c nvidia faiss-gpu=1.8.0 pytorch="*=cuda" pytorch-cuda=12 numpy
It works.