OpenMined/PSI

Building a Python Wheel (on Ubuntu) with lower version of Protobuf

Closed this issue · 25 comments

chesterxgchen commented

Question

What's the approach to build Python wheel with lower version of Protobuf ?

Further Information

While trying to integrate with Openmined PSI with NVFLARE, our CI/CD process reported that

openmined-psi 1.0.0 depends on protobuf==4.21.12
    nvflare-nightly 0+untagged.1.g34f97a7 depends on protobuf==3.20.2

So we hope to build a wheel that has lower protobuf version requirement.

Look at the code, we changed the following files

https://github.com/OpenMined/PSI/blob/master/private_set_intersection/python/requirements.in#L7
https://github.com/OpenMined/PSI/blob/master/private_set_intersection/python/BUILD#L105

and use

pip-compile requirements.in

to generate the requirements.txt

then use the following command to build the wheel:

bazel build -c opt //private_set_intersection/python:wheel

The output showed the following warning:

~/projects/PSI/private_set_intersection/python$ bazel build -c opt //private_set_intersection/python:wheel
INFO: Analyzed target //private_set_intersection/python:wheel (135 packages loaded, 2284 targets configured).
INFO: Found 1 target...
INFO: From Compiling src/google/protobuf/message_lite.cc [for tool]:
In file included from /usr/include/string.h:535,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/port.h:39,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/common.h:48,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.h:46,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:36:
In function 'void* memcpy(void*, const void*, size_t)',
    inlined from 'uint8_t* google::protobuf::io::EpsCopyOutputStream::WriteRaw(const void*, int, uint8_t*)' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/io/coded_stream.h:684:16,
    inlined from 'virtual uint8_t* google::protobuf::internal::ImplicitWeakMessage::_InternalSerialize(uint8_t*, google::protobuf::io::EpsCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/implicit_weak_message.h:103:28,
    inlined from 'bool google::protobuf::MessageLite::SerializePartialToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:411:30:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: warning: 'void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)' specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   29 |   return __builtin___memcpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   30 |                                  __glibc_objsize0 (__dest));
      |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~
Target //private_set_intersection/python:wheel up-to-date:
  bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl
INFO: Elapsed time: 27.419s, Critical Path: 11.53s
INFO: 737 processes: 27 internal, 709 linux-sandbox, 1 local.
INFO: Build completed successfully, 737 total actions

Although it said its successful, but there is no wheel

~/projects/PSI/private_set_intersection/python$ ls -l bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl
ls: cannot access 'bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl': No such file or directory

Screenshots

image

System Information

  • OS: Ubuntu x86_64
  • OS Version: 20.04
  • Language Version: python 3.8.16
  • Package Manager Version:, bazel 6.0.0

Additional Context

This is part of the effort trying to integrate openmined PSI with NVFLARE in Federated learning. Although with protobuf version conlicts. My local tests seems to be fine. But the unit tests failed in CI process. So we are hoping there is a build with lower protobuf requirements.

Does the PSI python needs such the latest version of protobuf ?

s0l0ist commented

@chesterxgchen, thanks for raising this. The steps you've taken to lower the protobuf requirement are correct. The requirement is only needed at runtime and isn't used for building the wheel so lowering it should just work. I haven't tested the wheel explicitly to confirm, however.

Regarding the wheel output, the file is located in the parent directory. You're using ls in the private_set_intersection/python directory and need to be in the root (two levels up). If you run thebazel commands from the root of the repo, this helps prevent confusion on where to find artifacts.

image

Please note there is a script to rename the wheel in CI_CD.yml to correct the ABI and INTERPRETER placeholders (if you're intending to publish it privately)

chesterxgchen commented

Thanks for the quick response, let me give it a try

chesterxgchen commented

@s0l0ist Nick, I did find the wheel, I built.
The original ticket could be closed.

But there is an related issues,
I ran the rename.py script and then installed the wheel via pip iinstall, but there is one issue.

  • the openmined_psi module is not packaged in the wheel.
    Here are the output:
    image

Once I unzip the wheel, I did not find the module either:

image

chesterxgchen commented

BTW, I did the similar built in Mac M1, saw the same thing. It might be something I did not do correctly.

s0l0ist commented

Ah yeah that was a minor breaking change with the 1.0.0 release for Python. The import paths have changed. Please use something like import private_set_intersection.python as psi. The python tests have examples.

chesterxgchen commented

still show errors

(nvflare-env2) chester@RTX:~/projects/PSI$ python
Python 3.8.16 (default, Dec  7 2022, 01:12:06) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import private_set_intersection.python as psi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chester/projects/PSI/private_set_intersection/python/__init__.py", line 6, in <module>
    from private_set_intersection.python import _openmined_psi as psi
ImportError: cannot import name '_openmined_psi' from partially initialized module 'private_set_intersection.python' (most likely due to a circular import) (/home/chester/projects/PSI/private_set_intersection/python/__init__.py)
s0l0ist commented

Hmm, try uninstalling from pip, then bazel clean then try building and installing the wheel. I don't have a linux machine to test at the moment.

chesterxgchen commented

I did the pip uninstall, bazel clean and bazel build. Still get the same error

$ pip uninstall openmined.psi==1.0.0
Found existing installation: openmined.psi 1.0.0
Uninstalling openmined.psi-1.0.0:
  Would remove:
    /home/chester/nvflare-env2/lib/python3.8/site-packages/openmined.psi-1.0.0.dist-info/*
    /home/chester/nvflare-env2/lib/python3.8/site-packages/private_set_intersection/proto/psi_python_proto_pb/private_set_intersection/proto/psi_pb2.py
    /home/chester/nvflare-env2/lib/python3.8/site-packages/private_set_intersection/python/*
    /home/chester/nvflare-env2/lib/python3.8/site-packages/python/google/protobuf/*
Proceed (Y/n)? Y
  Successfully uninstalled openmined.psi-1.0.0


$ bazel clean
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel build -c opt //private_set_intersection/python:wheel
INFO: Analyzed target //private_set_intersection/python:wheel (135 packages loaded, 2284 targets configured).
INFO: Found 1 target...
INFO: From Compiling src/google/protobuf/message_lite.cc [for tool]:
In file included from /usr/include/string.h:535,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/port.h:39,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/common.h:48,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.h:46,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:36:
In function 'void* memcpy(void*, const void*, size_t)',
    inlined from 'uint8_t* google::protobuf::io::EpsCopyOutputStream::WriteRaw(const void*, int, uint8_t*)' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/io/coded_stream.h:684:16,
    inlined from 'virtual uint8_t* google::protobuf::internal::ImplicitWeakMessage::_InternalSerialize(uint8_t*, google::protobuf::io::EpsCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/implicit_weak_message.h:103:28,
    inlined from 'bool google::protobuf::MessageLite::SerializePartialToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:411:30:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: warning: 'void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)' specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   29 |   return __builtin___memcpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   30 |                                  __glibc_objsize0 (__dest));
      |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~
Target //private_set_intersection/python:wheel up-to-date:
  bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl
INFO: Elapsed time: 27.351s, Critical Path: 11.41s
INFO: 737 processes: 27 internal, 709 linux-sandbox, 1 local.
INFO: Build completed successfully, 737 total actions

$ python private_set_intersection/python/rename.py 
renaming  ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-cp38-cp38-manylinux2014_x86_64.whl

$ pip install ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-cp38-cp38-manylinux2014_x86_64.whl
Processing ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-cp38-cp38-manylinux2014_x86_64.whl
Requirement already satisfied: protobuf==3.20.2 in /home/chester/nvflare-env2/lib/python3.8/site-packages (from openmined.psi==1.0.0) (3.20.2)
Installing collected packages: openmined.psi
Successfully installed openmined.psi-1.0.0

$ python 
Python 3.8.16 (default, Dec  7 2022, 01:12:06) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import private_set_intersection.python as psi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chester/projects/PSI/private_set_intersection/python/__init__.py", line 6, in <module>
    from private_set_intersection.python import _openmined_psi as psi
ImportError: cannot import name '_openmined_psi' from partially initialized module 'private_set_intersection.python' (most likely due to a circular import) (/home/chester/projects/PSI/private_set_intersection/python/__init__.py)
>>>
  • 😕1
s0l0ist commented

I haven't been able to reproduce your exact error, but I did encounter something strange when using pyenv to switch down to 3.8.16. The wheel appeared to build successfully, but would also fail to import because of an unsupported pybind11 function that was added in 3.9+. To fix, I needed to clear the bazel cache located in the home directory of the active user rm -rf ~/.cache. Then I was able to successfully re-run the tests and build & install the wheel.

  1. Navigate to the root of the repository:cd PSI
  2. bazel clean --expunge
  3. rm -rf ~/.cache
  4. bazel build -c opt //private_set_intersection/python:wheel
  5. python private_set_intersection/python/rename.py
  6. pip install ...
  7. <run REPL or test script>

I'm more inclined to believe this is a bazel / system issue as the library gets tested and published using 3.8 + ubuntu in CI_CD.

  • 👍1
chesterxgchen commented

Thanks for the detailed steps, There is one issue I encountered earlier, but I managed to fake it. I forgot about this when I open this issue. By following your steps, this issue came back.

$ bazel clean --expunge
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.


$ rm -rf ~/.cache

$ bazel build -c opt //private_set_intersection/python:wheel
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Repository local_config_python instantiated at:
  /home/chester/projects/PSI/WORKSPACE:11:9: in <toplevel>
  /home/chester/projects/PSI/private_set_intersection/deps.bzl:108:21: in psi_deps
Repository rule python_configure defined at:
  /home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl:412:35: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_python':
   Traceback (most recent call last):
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 409, column 36, in _python_autoconf_impl
		_create_local_python_repository(repository_ctx)
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 360, column 51, in _create_local_python_repository
		python_include_rule = _symlink_genrule_for_dir(
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 131, column 43, in _symlink_genrule_for_dir
		files = "\n".join(sorted(_read_dir(repository_ctx, src_dir).splitlines()))
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 82, column 31, in _read_dir
		find_result = _execute(
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 56, column 14, in _execute
		_fail("\n".join([
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 27, column 9, in _fail
		fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Python Configuration Error: Repository command failed
find: ‘/usr/include/python3.8’: No such file or directory

ERROR: /home/chester/projects/PSI/WORKSPACE:11:9: fetching python_configure rule //external:local_config_python: Traceback (most recent call last):
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 409, column 36, in _python_autoconf_impl
		_create_local_python_repository(repository_ctx)
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 360, column 51, in _create_local_python_repository
		python_include_rule = _symlink_genrule_for_dir(
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 131, column 43, in _symlink_genrule_for_dir
		files = "\n".join(sorted(_read_dir(repository_ctx, src_dir).splitlines()))
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 82, column 31, in _read_dir
		find_result = _execute(
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 56, column 14, in _execute
		_fail("\n".join([
	File "/home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/pybind11_bazel/python_configure.bzl", line 27, column 9, in _fail
		fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Python Configuration Error: Repository command failed
find: ‘/usr/include/python3.8’: No such file or directory

INFO: Repository com_google_absl instantiated at:
  /home/chester/projects/PSI/WORKSPACE:11:9: in <toplevel>
  /home/chester/projects/PSI/private_set_intersection/deps.bzl:48:21: in psi_deps
Repository rule http_archive defined at:
  /home/chester/.cache/bazel/_bazel_chester/954be5a46403ba372c0eb05f16a409f0/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
ERROR: /home/chester/projects/PSI/private_set_intersection/python/BUILD:8:17: //private_set_intersection/python:_openmined_psi.so depends on @local_config_python//:python_headers in repository @local_config_python which failed to fetch. no such package '@local_config_python//': Python Configuration Error: Repository command failed
find: ‘/usr/include/python3.8’: No such file or directory

ERROR: Analysis of target '//private_set_intersection/python:wheel' failed; build aborted: 
INFO: Elapsed time: 18.893s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (110 packages loaded, 1180 targets configured)

$ which python
/home/chester/nvflare-env2/bin/python
$ python --version
Python 3.8.16

I have venv with python 3.8.16, but I don't have
‘/usr/include/python3.8’

although I do have /usr/include/python3.10/

I remember at one point, I kind of fake" it, by setup a symbolic link to /usr/include/python3.10/ as /usr/include/python3.8/

**is ‘/usr/include/python3.8’ ** really required ?

s0l0ist commented

It appears that the bazel rule for pybind11 requires a local python to be installed and available. I do not have more insight on this issue. I recall at some point in time needing to install sudo apt install python3-dev on Ubuntu 22.04.

chesterxgchen commented

thanks for the tip, I will give it a try: sudo apt install python3-dev

chesterxgchen commented

Looks like I already have python3-dev installed. I removed it and re-installed again,

$ sudo apt install python3-dev
....
Preparing to unpack .../python3-dev_3.10.6-1~22.04_amd64.deb ...
Unpacking python3-dev (3.10.6-1~22.04) ...
Setting up python3-dev (3.10.6-1~22.04) ...
Processing triggers for man-db (2.10.2-1) ...

Noticed that its unpacking the python 3.10.6. so I did again with the following

sudo apt install python3.8-dev

This time, it worked, it created the /usr/include/python3.8/ directory.

I repeat your 7 steps, the build was successful, but the last steps it still not working

 bazel clean --expunge
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ rm -rf ~/.cache

$ bazel build -c opt //private_set_intersection/python:wheel
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //private_set_intersection/python:wheel (135 packages loaded, 2284 targets configured).
INFO: Found 1 target...
INFO: From Compiling src/google/protobuf/message_lite.cc [for tool]:
In file included from /usr/include/string.h:535,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/port.h:39,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/common.h:48,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.h:46,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:36:
In function 'void* memcpy(void*, const void*, size_t)',
    inlined from 'uint8_t* google::protobuf::io::EpsCopyOutputStream::WriteRaw(const void*, int, uint8_t*)' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/io/coded_stream.h:684:16,
    inlined from 'virtual uint8_t* google::protobuf::internal::ImplicitWeakMessage::_InternalSerialize(uint8_t*, google::protobuf::io::EpsCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/implicit_weak_message.h:103:28,
    inlined from 'bool google::protobuf::MessageLite::SerializePartialToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:411:30:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: warning: 'void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)' specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   29 |   return __builtin___memcpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   30 |                                  __glibc_objsize0 (__dest));
      |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~
Target //private_set_intersection/python:wheel up-to-date:
  bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl
INFO: Elapsed time: 61.337s, Critical Path: 11.61s
INFO: 737 processes: 27 internal, 709 linux-sandbox, 1 local.
INFO: Build completed successfully, 737 total actions

$ python private_set_intersection/python/rename.py
renaming  ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-cp38-cp38-manylinux2014_x86_64.whl

$ pip install ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-cp38-cp38-manylinux2014_x86_64.whl
Processing ./bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-cp38-cp38-manylinux2014_x86_64.whl
Requirement already satisfied: protobuf==3.20.2 in /home/chester/nvflare-env2/lib/python3.8/site-packages (from openmined.psi==1.0.0) (3.20.2)
Installing collected packages: openmined.psi
Successfully installed openmined.psi-1.0.0

$ python
Python 3.8.16 (default, Dec  7 2022, 01:12:06) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import private_set_intersection.python as psi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chester/projects/PSI/private_set_intersection/python/__init__.py", line 6, in <module>
    from private_set_intersection.python import _openmined_psi as psi
ImportError: cannot import name '_openmined_psi' from partially initialized module 'private_set_intersection.python' (most likely due to a circular import) (/home/chester/projects/PSI/private_set_intersection/python/__init__.py)
>>>
s0l0ist commented

I don't know how to fix this issue. It still feels like a system / python issue.

What works for me on Ubuntu 22.04 is the following:

  1. Install pyenv
  2. Install the version of python via pyenv install 3.8.16
  3. Set it to the global interpreter pyenv global 3.8.16
  4. Run the bazel cleanup steps to purge everything. Only need to do this once.
  5. Run the bazel commands.
chesterxgchen commented

I was using venv let me try your approach

s0l0ist commented

I'll also relax the protobuf requirements as well.

  • 👍1
  • ❤️1
chesterxgchen commented

I followed your approach and did the followings, unfortunately, I still get the same errors.

  1. deactivate ( so I am no longer in venv)
  2. installed the pyenv
  3. pyenv install 3.8.16
    I have to do this many times, due to missing needed dependencies
  4. went through the rest of build steps, and got the wheel.
  5. I did the pip install in both venv and non-venv.
  6. in REPL, same error
  • 😕1
s0l0ist commented

@chesterxgchen 1.0.2 has been published.

  • ❤️1
chesterxgchen commented

thank you so much

chesterxgchen commented

I still having the same error, let me ask my colleague to try this from another computer

pip install openmined.psi==1.0.2
Collecting openmined.psi==1.0.2
  Using cached openmined.psi-1.0.2-cp38-cp38-manylinux2014_x86_64.whl (2.0 MB)
Requirement already satisfied: protobuf>=3.20 in /home/chester/.pyenv/versions/3.8.16/lib/python3.8/site-packages (from openmined.psi==1.0.2) (3.20.2)
Installing collected packages: openmined.psi
Successfully installed openmined.psi-1.0.2
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/home/chester/.pyenv/versions/3.8.16/bin/python3.8 -m pip install --upgrade pip' command.

$ python
Python 3.8.16 (default, Jan  6 2023, 11:28:23) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from private_set_intersection.python import _openmined_psi as psi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chester/projects/PSI/private_set_intersection/python/__init__.py", line 6, in <module>
    from private_set_intersection.python import _openmined_psi as psi
ImportError: cannot import name '_openmined_psi' from partially initialized module 'private_set_intersection.python' (most likely due to a circular import) (/home/chester/projects/PSI/private_set_intersection/python/__init__.py)
>>>
chesterxgchen commented

@s0l0ist Nick, I just asked my colleague to pip install the latest release and try this out

>>> import private_set_intersection.python as psi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/__init__.py", line 6, in <module>
    from private_set_intersection.python import _openmined_psi as psi
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so)
>>> 
[0] 0:python3.10*

He pointed out that

ubuntu 20.04 only has glibc 2.31 (https://launchpad.net/ubuntu/+source/glibc).

_openmined_psi.so was built with 2.33 gblic.

chesterxgchen commented

My colleague gave me the followings

$ ldd /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so
/home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so)
/home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so)
/home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so)
/home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so)
/home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/iscyang/.local/share/virtualenvs/NVFlare-mDS0_8dN/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so)
        linux-vdso.so.1 (0x00007ffeb961e000)
        libgtk3-nocsd.so.0 => /lib/x86_64-linux-gnu/libgtk3-nocsd.so.0 (0x00007f4ccb175000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4ccb026000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4ccae44000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4ccae29000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4ccac37000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4ccb733000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4ccac31000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4ccac0c000)

and I did same,but have different output

ldd /home/chester/nvflare-env2/lib/python3.8/site-packages/private_set_intersection/python/_openmined_psi.so
	linux-vdso.so.1 (0x00007fff516c6000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f74d5777000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f74d554d000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f74d552d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f74d5305000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f74d5c0d000)

Is it correct that
linux-vdso.so.1 (0x00007fff516c6000)

can't be resolved in both case ?

s0l0ist commented

Aha! Thanks for digging. We publish the wheels on Ubuntu 22, I need to update the job to build on ubuntu 20.04 and change the manylinux2014 attribute depending on the OS/Glibc.

This will require a few changes to CI and to the bazel wheel rule.

  • 👍1
s0l0ist commented

Should be fixed with #150

chesterxgchen commented

I installed 1.0.3 version, it seems to work. many thanks

  • ❤️1