Strange import behavior, OSError libllvmlite.so

[detkov]

Prerequisites:

> pip list
nnabla                                1.29.0
nnabla-converter                      1.29.0
nnabla-ext-cuda110                    1.29.0

> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

> python -V
Python 3.7.12

> nvidia-smi
NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2

I have the code importing nnabla together with nnabla.ext_utils, some other libs and module (some_nnabla_module), inside of which there also is nnabla importing.

Imports of the file.py I run via python file.py:

import argparse
import os
import shutil
import warnings
from glob import glob
from os.path import basename, join, splitext

import nnabla
import numpy as np
from nnabla.ext_utils import get_extension_context
from PIL import Image

from some_nnabla_module import some_nnabla_method
from some_module import some_method

In case it is needed, imports of the some_nnabla_module file:

import nnabla
import nnabla.functions as F
import numpy as np
import yaml
from nnabla.utils.image_utils import imread
from PIL import Image
from scipy.spatial import ConvexHull

And when I run python file.py, I get the following trace:

[nnabla][INFO]: Initializing CPU extension...
Traceback (most recent call last):
  File "file.py", line 14, in <module>
    from some_module import some_method
  File "/home/nikita/some_dir/some_module.py", line 9, in <module>
    import librosa
  File "/opt/conda/lib/python3.7/site-packages/librosa/__init__.py", line 209, in <module>
    from . import core
  File "/opt/conda/lib/python3.7/site-packages/librosa/core/__init__.py", line 5, in <module>
    from .convert import *  # pylint: disable=wildcard-import
  File "/opt/conda/lib/python3.7/site-packages/librosa/core/convert.py", line 7, in <module>
    from . import notation
  File "/opt/conda/lib/python3.7/site-packages/librosa/core/notation.py", line 8, in <module>
    from ..util.exceptions import ParameterError
  File "/opt/conda/lib/python3.7/site-packages/librosa/util/__init__.py", line 77, in <module>
    from .utils import *  # pylint: disable=wildcard-import
  File "/opt/conda/lib/python3.7/site-packages/librosa/util/utils.py", line 9, in <module>
    import numba
  File "/opt/conda/lib/python3.7/site-packages/numba/__init__.py", line 19, in <module>
    from numba.core import config
  File "/opt/conda/lib/python3.7/site-packages/numba/core/config.py", line 15, in <module>
    import llvmlite.binding as ll
  File "/opt/conda/lib/python3.7/site-packages/llvmlite/binding/__init__.py", line 4, in <module>
    from .dylib import *
  File "/opt/conda/lib/python3.7/site-packages/llvmlite/binding/dylib.py", line 3, in <module>
    from llvmlite.binding import ffi
  File "/opt/conda/lib/python3.7/site-packages/llvmlite/binding/ffi.py", line 195, in <module>
    raise OSError(msg)
OSError: Could not load shared object file: libllvmlite.so
Errors were: [OSError("/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /opt/conda/lib/python3.7/site-packages/llvmlite/binding/../../../../libLLVM-11.so)"), OSError('libllvmlite.so: cannot open shared object file: No such file or directory'), OSError('./libllvmlite.so: cannot open shared object file: No such file or directory'), OSError("/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /opt/conda/lib/python3.7/site-packages/llvmlite/binding/../../../../libLLVM-11.so)")]

Strange, isn't it? But if we move importing of nnabla and nnabla.ext_utils after numpy and PIL in file.py, like this:

import nnabla
import numpy as np
from nnabla.ext_utils import get_extension_context
from PIL import Image
--->
import numpy as np
from PIL import Image
import nnabla
from nnabla.ext_utils import get_extension_context

After this, the error is not showing. Any ideas on what this might be?

[TomonobuTsujikawa]

I tried a bit, and there is no problem.

libnnabla.so also has a dependency to libstdc++.so, but I think it might be conda related issue.
because this path /opt/conda/lib/python3.7/site-packages/llvmlite/binding/../../../../libLLVM-11.so is points to /opt/conda/lib/.

In my environment, llvmlite/binding has libllvmlite.so, and this file seems to have correct dependency.

So, you can check which glibcxx is supported by following command: strings libstdc++.so.6 | grep GLIBCXX
And, What dependencies each library has: ldd libLLVM-11.so

some users could resolve this issue by conda install libgcc, or apt-get upgrade libstdc++6,
but I think it is better to check conda environment and llvmlite at first.