specify GLIBC version?
jarussell opened this issue · 9 comments
I am building on a linux/aarch64 docker image for an M1 Mac based on debian bullseye. When building with a self-compiled version of tensorflow on aarch64, I get a successful build for both tensorflow and tensorflow-text. However, bazel downloads and uses a separate version of GLIBC than what I have on my system for building tensorflow-text, resulting in an unusable package for tensorflow-text.
Using bazel 5.1.1 and tensorflow 2.9.0, I get numerous errors like:
E OSError: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/local/lib/python3.10/site-packages/tensorflow/python/platform/../../core/platform/_cpu_feature_guard.so)
___________________________________________________________________ ERROR collecting service/text/tensorflow_text/tools/wordpiece_vocab/bert_vocab_from_dataset_test.py ____________________________________________________________________
/usr/local/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
???
<frozen importlib._bootstrap>:1027: in _find_and_load
???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1050: in _gcd_import
???
<frozen importlib._bootstrap>:1027: in _find_and_load
???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1050: in _gcd_import
???
<frozen importlib._bootstrap>:1027: in _find_and_load
???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1050: in _gcd_import
???
<frozen importlib._bootstrap>:1027: in _find_and_load
???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:688: in _load_unlocked
???
<frozen importlib._bootstrap_external>:883: in exec_module
???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
???
text/tensorflow_text/__init__.py:17: in <module>
from tensorflow.python.util.all_util import remove_undocumented
/usr/local/lib/python3.10/site-packages/tensorflow/__init__.py:37: in <module>
from tensorflow.python.tools import module_util as _module_util
/usr/local/lib/python3.10/site-packages/tensorflow/python/__init__.py:36: in <module>
from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
/usr/local/lib/python3.10/site-packages/tensorflow/python/pywrap_tensorflow.py:26: in <module>
self_check.preload_check()
/usr/local/lib/python3.10/site-packages/tensorflow/python/platform/self_check.py:65: in preload_check
ctypes.CDLL(cpu_feature_guard_library)
/usr/local/lib/python3.10/ctypes/__init__.py:374: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/local/lib/python3.10/site-packages/tensorflow/python/platform/../../core/platform/_cpu_feature_guard.so)
____________________________________________________________________________ ERROR collecting service/text/tensorflow_text/tools/wordpiece_vocab/utils_test.py
I am not sure whether this is a bazel, tensorflow, or tensorflow-text issue. However, the instructions and build system for tensorflow-text are the hardest to follow. On tensorflow I can simply run ./configure to set my architecture and let it autodetect everything else, building with -march=native. oss_scripts/run_build.sh does not seem to detect the same version of GLIBC. Latest GLIBC on debian bullseye is 2.3.1 while tensorflow-text uses 2.3.2.
How do I tell tensorflow-text to use my GLIBC instead of some other build tools?
We try to remove all configuration by delegating to the .bazelrc & .bazelversion in core TF. However, there are some edits made by their configure script, though we believe these to be unrelated to the TF Text build. Can you compare what has changed when you run the configure script on a clean checkout of TF?
I'm not 100% sure what you're asking. It looks like tensorflow-text downloads the bazel build configuration from the latest tensorflow. My script checks out tensorflow==2.9.0, builds it with -march=native -Wno-sign-compare and uploads it to a Nexus host. tensorflow-text then immediately installs tensorflow from that Nexus host right after.
I am not sure if I have to export variables for the proper detection of my compiler or whether bazel uses ldconfig or ldd to set its variables. The way the build is set up with the text/oss_scripts/run_build.sh running configure on its own, makes it hard to pass in the same flags that I would pass into configure for the regular tensorflow build to ensure they get the same flags.
Are you asking for differences between the tensorflow .bazelrc and the tensorflow-text .bazelrc for my build?
diff /tmp/text/.bazelrc /tmp/tensorflow/.bazelrc
24a25,28
> # c++17: Build with C++17 options (links with libc++)
> # c++1z: Build with C++17 options (links with libc++)
> # c++17_gcc: Build with C++17 options (links with stdlibc++)
> # c++1z_gcc: Build with C++17 options (links with stdlibc++)
126c130
<
---
> build --noincompatible_remove_legacy_whole_archive
148,150d151
< # cc_shared_library ensures no library is linked statically more than once.
< build --experimental_link_static_libraries_once=false
<
188d188
< build:macos_arm64 --macos_minimum_os=11.0
198,199d197
< build:ios_sim_arm64 --config=ios
< build:ios_sim_arm64 --cpu=ios_sim_arm64
212d209
< build:monolithic --experimental_link_static_libraries_once=false # b/229868128
231d227
< build:mkl_aarch64 --define=build_with_acl=true
234,238d229
< # Config setting to build oneDNN with Compute Library for the Arm Architecture (ACL).
< # with Eigen threadpool support
< build:mkl_aarch64_threadpool --define=build_with_mkl_aarch64=true
< build:mkl_aarch64_threadpool -c opt
<
273d263
< build:rocm --experimental_link_static_libraries_once=false # b/230048163
289a280,286
> # Build TF with C++ 17 features.
> build:c++17 --cxxopt=-std=c++1z
> build:c++17 --cxxopt=-stdlib=libc++
> build:c++1z --config=c++17
> build:c++17_gcc --cxxopt=-std=c++1z
> build:c++1z_gcc --config=c++17_gcc
>
301,307d297
< build:windows --host_copt=/W0
<
<
< # On Windows, `__cplusplus` is wrongly defined without this switch
< # See https://devblogs.microsoft.com/cppblog/msvc-now-correctly-reports-__cplusplus/
< build:windows --copt=/Zc:__cplusplus
< build:windows --host_copt=/Zc:__cplusplus
314,322d303
< # Windows has a relatively short command line limit, which TF has begun to hit.
< # See https://docs.bazel.build/versions/main/windows.html
< build:windows --features=compiler_param_file
<
< # Speed Windows compile times. Available in VS 16.4 (we are on 16.11). See
< # https://groups.google.com/a/tensorflow.org/d/topic/build/SsW98Eo7l3o/discussion
< build:windows --copt=/d2ReducedOptimizeHugeFunctions
< build:windows --host_copt=/d2ReducedOptimizeHugeFunctions
<
334,344c315,325
< # By default, build TF in C++ 17 mode.
< build:android --cxxopt=-std=c++17
< build:android --host_cxxopt=-std=c++17
< build:ios --cxxopt=-std=c++17
< build:ios --host_cxxopt=-std=c++17
< build:linux --cxxopt=-std=c++17
< build:linux --host_cxxopt=-std=c++17
< build:macos --cxxopt=-std=c++17
< build:macos --host_cxxopt=-std=c++17
< build:windows --cxxopt=/std:c++17
< build:windows --host_cxxopt=/std:c++17
---
> # By default, build TF in C++ 14 mode.
> build:android --cxxopt=-std=c++14
> build:android --host_cxxopt=-std=c++14
> build:ios --cxxopt=-std=c++14
> build:ios --host_cxxopt=-std=c++14
> build:linux --cxxopt=-std=c++14
> build:linux --host_cxxopt=-std=c++14
> build:macos --cxxopt=-std=c++14
> build:macos --host_cxxopt=-std=c++14
> build:windows --cxxopt=/std:c++14
> build:windows --host_cxxopt=/std:c++14
502,513c483,494
< build:rbe_linux_cuda_clang_base --crosstool_top="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain"
< build:rbe_linux_cuda_clang_base --extra_toolchains="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain-linux-x86_64"
< build:rbe_linux_cuda_clang_base --extra_execution_platforms="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_platform//:platform"
< build:rbe_linux_cuda_clang_base --host_platform="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_platform//:platform"
< build:rbe_linux_cuda_clang_base --platforms="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_platform//:platform"
< build:rbe_linux_cuda_clang_base --repo_env=TF_CUDA_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda"
< build:rbe_linux_cuda_clang_base --repo_env=TF_TENSORRT_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_tensorrt"
< build:rbe_linux_cuda_clang_base --repo_env=TF_NCCL_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_nccl"
< build:rbe_linux_cuda_clang_py37 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.7"
< build:rbe_linux_cuda_clang_py38 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.8"
< build:rbe_linux_cuda_clang_py39 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.9"
< build:rbe_linux_cuda_clang_py310 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.10"
---
> build:rbe_linux_cuda_clang_base --crosstool_top="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain"
> build:rbe_linux_cuda_clang_base --extra_toolchains="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain-linux-x86_64"
> build:rbe_linux_cuda_clang_base --extra_execution_platforms="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_platform//:platform"
> build:rbe_linux_cuda_clang_base --host_platform="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_platform//:platform"
> build:rbe_linux_cuda_clang_base --platforms="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_platform//:platform"
> build:rbe_linux_cuda_clang_base --repo_env=TF_CUDA_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda"
> build:rbe_linux_cuda_clang_base --repo_env=TF_TENSORRT_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_tensorrt"
> build:rbe_linux_cuda_clang_base --repo_env=TF_NCCL_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_nccl"
> build:rbe_linux_cuda_clang_py37 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.7"
> build:rbe_linux_cuda_clang_py38 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.8"
> build:rbe_linux_cuda_clang_py39 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.9"
> build:rbe_linux_cuda_clang_py310 --config=rbe_linux_cuda_clang_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu18.04-clang_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_python3.10"
537,538c518,519
< build:rbe_win --crosstool_top="//tensorflow/tools/toolchains/win/tf_win_06152022:toolchain"
< build:rbe_win --extra_toolchains="//tensorflow/tools/toolchains/win/tf_win_06152022:cc-toolchain-x64_windows"
---
> build:rbe_win --crosstool_top="//tensorflow/tools/toolchains/win/tf_win_02212022:toolchain"
> build:rbe_win --extra_toolchains="//tensorflow/tools/toolchains/win/tf_win_02212022:cc-toolchain-x64_windows"
629c610
< build:release_gpu_linux --action_env=LD_LIBRARY_PATH="/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-11.1/lib64:/usr/local/tensorrt/lib"
---
> build:release_gpu_linux --action_env=LD_LIBRARY_PATH="/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/tensorrt/lib"
632a614,619
> build:release_gpu_linux_11_4 --config=release_gpu_linux
> build:release_gpu_linux_11_4 --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda-11.4"
> build:release_gpu_linux_11_4 --action_env=TF_CUDA_VERSION="11.4"
> build:release_gpu_linux_11_4 --action_env=TF_CUDNN_VERSION="8.2"
> build:release_gpu_linux_11_4 --crosstool_top=@ubuntu18.04-gcc7_manylinux2010-cuda11.4-cudnn8.2-tensorrt7.2_config_cuda//crosstool:toolchain
>
635a623,625
> # First available in VS 16.4. Speeds Windows compile times by a lot. See
> # https://groups.google.com/a/tensorflow.org/d/topic/build/SsW98Eo7l3o/discussion
> build:release_cpu_windows --copt=/d2ReducedOptimizeHugeFunctions --host_copt=/d2ReducedOptimizeHugeFunctions
672c662
< build --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
---
> build --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
674,679c664,683
< build:manylinux2010 --config=release_cpu_linux
< build:manylinux2014 --config=release_cpu_linux
< build --action_env TF_HEADER_DIR="/usr/local/lib/python3.10/site-packages/tensorflow/include"
< build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.10/site-packages/tensorflow"
< build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
< build --action_env TF_CXX11_ABI_FLAG="1"
---
>
> # Experimental configuration for building XLA GPU lowering to TFRT.
> build:experimental_enable_xlir --config=tfrt
> build:experimental_enable_xlir --@tf_runtime//:enable_gpu
> build:experimental_enable_xlir --@rules_cuda//cuda:cuda_runtime=//tensorflow/compiler/xla/service/gpu:cuda_runtime_for_xlir
> build:experimental_enable_xlir --nocheck_visibility
> build:experimental_enable_xlir --incompatible_strict_action_env
> build:experimental_enable_xlir --config=monolithic
> build:experimental_enable_xlir --//tensorflow/compiler/xla/service/gpu:enable_xlir
>
> # bazel test --config=experimental_enable_bef_thunk \
> # //tensorflow/compiler/xla/service/gpu:bef_thunk_tests
> build:experimental_enable_bef_thunk --config=experimental_enable_xlir
> test:experimental_enable_bef_thunk --test_env=XLA_FLAGS=--xla_gpu_bef_thunk
>
> # bazel test --config=experimental_enable_bef_executable \
> # //tensorflow/compiler/xla/service/gpu:bef_executable_tests
> build:experimental_enable_bef_executable --config=experimental_enable_xlir
> test:experimental_enable_bef_executable --test_env=XLA_FLAGS=--xla_gpu_bef_executable
>
I am missing something and the build instructions do not cover what I am missing. On arm64/aarch64 I am trying to compile both tensorflow==2.9.0 and tensorflow-text so that they can work together in my Docker container because there is no official release for arm64/aarch64.
I compile tensorflow with:
git clone --depth 1 --branch "v2.9.0" --single-branch https://github.com/tensorflow/tensorflow.git
pushd tensorflow
# configure questions asked in order:
#Please specify the location of python. [Default is /usr/local/bin/python3]:
#Please input the desired Python library path to use. Default is [/orm/]
#Do you wish to build TensorFlow with ROCm support? [y/N]:
#Do you wish to build TensorFlow with CUDA support? [y/N]:
#Do you wish to download a fresh release of clang? (Experimental) [y/N]:
#Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:
#Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
echo -e "\n/usr/local/lib/python3.10/site-packages\nN\nN\nN\n-march=native -Wno-sign-compare\nN\n"|./configure
bazel build //tensorflow/tools/pip_package:build_pip_package
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
fname=$(find /tmp/tensorflow_pkg -type f -iname '*.whl'|grep -i tensorflow)
# upload to nexus so pip install in tensorflow-text will find it
[ -e "$fname" ] && twine upload "$fname" -r nexus
popd
Then for tensorflow-text, oss_scripts/install_bazel.sh searches for a .bazelrc but does not provide nor find one unless you manually installed bazel previously. So I manually install bazel, ensuring to grab the same version with the same curl command and add it to my $PATH. It is not clear to me why it even bothers searching for a .bazelrc if it's going to overwrite it and install a specific version anyways. At the very least, it doesn't make sense that it exits if it can't find it when the point was to install it.
# install bazel 5.1.1 and add to path
git clone https://github.com/tensorflow/text.git
pushd text
./oss_scripts/run_build.sh
popd
I am wondering what approach I need to take to fix this issue. I see a couple possible paths but am not super familiar with the customization of bazel. Possible approaches:
- Replace line 67 of configure.sh in TFT with
cp /tmp/tensorflow/.bazelrc /tmp/text/.bazelrcto use the same configure flags but allow TFT to override what it needs to. - Replace all of the configure script with multiple scripts to actually run configure with the correct flags in TFT.
- Change C/C++ flags (I don't know the right way to do this for
bazel). I was expectingbazelto detect my compiler usingldconfigbut it appears that I get different errors if I exportLD_LIBRARY_PATH,LIBRARY_PATH,CC, andCXXto my default system compiler.
I'm also wondering if there is a good way I can narrow down if this is a problem with my build of tensorflow or if it's a problem with my build of tensorflow-text.
It is not clear to me why it even bothers searching for a .bazelrc if it's going to overwrite it and install a specific version anyways. At the very least, it doesn't make sense that it exits if it can't find it when the point was to install it.
Running the configure script creates the .bazelrc file. You do not need to manually install Bazel.
Replace line 67 of configure.sh in TFT with cp /tmp/tensorflow/.bazelrc /tmp/text/.bazelrc to use the same configure flags but allow TFT to override what it needs to.
I would try this.
Making the modification to line 67 to copy the .bazelrc, I still had problems. I think part of the issue may be that TFT downloads the latest bazel from TF when that my not be what was expected for the release. Because of this, I switched to TF==2.9.1.
I get a warning (the build, itself, is successful):
...
writing manifest file 'tensorflow_text.egg-info/SOURCES.txt'
/usr/local/lib/python3.10/site-packages/setuptools/command/build_py.py:153: SetuptoolsDeprecationWarning: Installing 'tensorflow_text.core.pybinds' as data is deprecated, please list it in `packages`.
!!
############################
# Package would be ignored #
############################
Python recognizes 'tensorflow_text.core.pybinds' as an importable package,
but it is not listed in the `packages` configuration of setuptools.
'tensorflow_text.core.pybinds' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).
Please make sure that 'tensorflow_text.core.pybinds' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" and "data files" on setuptools
documentation page.
!!
check.warn(importable)
Which then results as an error when I try to import tensorflow_text:
import tensorflow_text
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import tensorflow_text
File /tmp/text/tensorflow_text/__init__.py:20, in <module>
17 from tensorflow.python.util.all_util import remove_undocumented
19 # pylint: disable=wildcard-import
---> 20 from tensorflow_text.core.pybinds import tflite_registrar
21 from tensorflow_text.python import keras
22 from tensorflow_text.python import metrics
ImportError: cannot import name 'tflite_registrar' from 'tensorflow_text.core.pybinds' (unknown location)
I think that I am close to having this running and I suspect this is something easy to fix with bazel, I am just not sure how to do it.
Edit: The warning was apparently not the error I am getting. I was trying to import from the directory I was building in. I still have an error when I import:
...
File /usr/local/lib/python3.10/site-packages/tensorflow_text/python/ops/__init__.py:57, in <module>
55 from tensorflow_text.python.ops.segment_combiner_ops import combine_segments
56 from tensorflow_text.python.ops.sentence_breaking_ops import sentence_fragments
---> 57 from tensorflow_text.python.ops.sentencepiece_tokenizer import SentencepieceTokenizer
58 from tensorflow_text.python.ops.sliding_window_op import sliding_window
59 from tensorflow_text.python.ops.split_merge_from_logits_tokenizer import SplitMergeFromLogitsTokenizer
File /usr/local/lib/python3.10/site-packages/tensorflow_text/python/ops/sentencepiece_tokenizer.py:29, in <module>
27 from tensorflow.python.ops.ragged import ragged_tensor
28 from tensorflow.python.ops.ragged.ragged_tensor import RaggedTensor
---> 29 from tensorflow.python.trackable import resource
30 from tensorflow_text.python.ops.tokenization import Detokenizer
31 from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
ModuleNotFoundError: No module named 'tensorflow.python.trackable'
This error appears to be because I cloned the latest tensorflow-text instead of the 2.9 version while building against the 2.9.x version of tensorflow. I am waiting on rebuilds for both TF and TFT before I know whether I finally got it to work.
Did it work out for you?
I am not sure. I currently have 3 remaining errors when running but I think they are numpy/tensorflow errors and not tensorflow-text errors. I will wait to close until I get it working in case I need to add more context. Thanks for your help so far!
The related error was a mismatched version of numpy with the 1.23.x version released, tensorflow needed a lower version to work.