pyinstaller fails with M1 MacOS tensorflow editions (tensorflow-macos and tensorflow-metal) - compiles CPU-only and ignores GPU
Closed this issue · 8 comments
Hi,
I'm trying to package an app on an M1 MacBook Pro using tensorflow-macos and tensorflow-metal - the editions of tensorflow for M1 Macs.
Contents of requirements.txt for pip:
tensorflow-macos==2.10.0
tensorflow-metal==0.6.0
tensorflow_addons==0.17.1
h5py==3.7.0
Pillow==9.2.0
tqdm==4.64.1
protobuf==3.19
ftfy==6.1.1
regex==2022.9.13
I am using plain python 3.10.7 - not using any of Conda/Anaconda/Microforge etc etc.
When the application is started using python interpreter, the GPU is recognised and utilised.
However when the application is built using pyinstaller, pyinstaller logs a series of errors about not being able to find "tensorflow" (see below) and the resulting artefact can only see the CPU - not the GPU - and so is an order of magnitude slower.
Build command:
pyinstaller sample-app.py --collect-all tensorflow --collect-all tensorflow-plugins --noconfirm --clean
I've attached a full log for completeness, but have also pasted a few relevant line extracts below.
Can anyone explain what's going on here and how to resolve the issue where pyinstaller is not able to handle tensorflow on M1 Mac's? I wondered if it might be linked to the existing tensorflow hook file hook-tensorflow.py but couldn't see documentation covering how these work.
77 INFO: UPX is not available.
...
83 WARNING: Unable to copy metadata for tensorflow: The 'tensorflow' distribution was not found and is required by the application
...
5198 WARNING: Unable to determine requirements for tensorflow: The 'tensorflow' distribution was not found and is required by the application
5204 WARNING: Unable to copy metadata for tensorflow-plugins: The 'tensorflow-plugins' distribution was not found and is required by the application
5330 WARNING: Unable to determine requirements for tensorflow-plugins: The 'tensorflow-plugins' distribution was not found and is required by the application
...
5545 INFO: Analyzing base_library.zip ...
5976 INFO: Loading module hook 'hook-heapq.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
6029 INFO: Loading module hook 'hook-encodings.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
6680 INFO: Loading module hook 'hook-pickle.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
7436 INFO: Caching module dependency graph...
7482 INFO: running Analysis Analysis-00.toc
7487 INFO: Analyzing /Users/samuel/git/stable-ui/backends/stable_tf/sample_app.py
7507 INFO: Loading module hook 'hook-numpy.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/numpy/_pyinstaller'...
7696 INFO: Loading module hook 'hook-numpy._pytesttester.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
7826 INFO: Loading module hook 'hook-difflib.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
7924 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
7990 INFO: Loading module hook 'hook-xml.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
8278 INFO: Loading module hook 'hook-platform.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
8413 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
9001 INFO: Processing pre-safe import module hook tensorflow from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-tensorflow.py'.
12489 INFO: Loading module hook 'hook-tensorflow.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
35294 INFO: Processing pre-find module path hook distutils from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
35436 INFO: Loading module hook 'hook-distutils.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
35437 INFO: Processing pre-find module path hook site from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks/pre_find_module_path/hook-site.py'.
35438 INFO: site: retargeting to fake-dir '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/fake-modules'
35553 INFO: Loading module hook 'hook-xml.dom.domreg.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
36142 INFO: Loading module hook 'hook-pkg_resources.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
36576 INFO: Processing pre-safe import module hook six.moves from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
38466 INFO: Loading module hook 'hook-packaging.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
40134 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-urllib3.packages.six.moves.py'.
40584 INFO: Loading module hook 'hook-certifi.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
43711 INFO: Loading module hook 'hook-h5py.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
44997 INFO: Loading module hook 'hook-grpc.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
47615 INFO: Loading module hook 'hook-PIL.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
47649 INFO: Loading module hook 'hook-PIL.Image.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
47970 INFO: Loading module hook 'hook-PIL.ImageFilter.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
52550 INFO: Loading module hook 'hook-wcwidth.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/PyInstaller/hooks'...
52595 INFO: Loading module hook 'hook-regex.py' from '/Users/samuel/git/stable-ui/backends/stable_tf/venv/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
52727 INFO: Analyzing hidden import 'tensorflow.__internal__'
52727 ERROR: Hidden import 'tensorflow.__internal__' not found
52727 INFO: Analyzing hidden import 'tensorflow.__internal__.autograph'
52727 ERROR: Hidden import 'tensorflow.__internal__.autograph' not found
52727 INFO: Analyzing hidden import 'tensorflow.__internal__.decorator'
52727 ERROR: Hidden import 'tensorflow.__internal__.decorator' not found
52727 INFO: Analyzing hidden import 'tensorflow.__internal__.dispatch'
52727 ERROR: Hidden import 'tensorflow.__internal__.dispatch' not found
52727 INFO: Analyzing hidden import 'tensorflow.__internal__.distribute'
52727 ERROR: Hidden import 'tensorflow.__internal__.distribute' not found
...
52728 INFO: Analyzing hidden import 'tensorflow._api.v2.compat.v1.compat.v1.estimator'
52728 ERROR: Hidden import 'tensorflow._api.v2.compat.v1.compat.v1.estimator' not found
52728 INFO: Analyzing hidden import 'tensorflow._api.v2.compat.v1.compat.v1.estimator.experimental'
52728 ERROR: Hidden import 'tensorflow._api.v2.compat.v1.compat.v1.estimator.experimental' not found
52728 INFO: Analyzing hidden import 'tensorflow._api.v2.compat.v1.compat.v1.estimator.export'
52729 ERROR: Hidden import 'tensorflow._api.v2.compat.v1.compat.v1.estimator.export' not found
...
52746 INFO: Analyzing hidden import 'tensorflow._api.v2.compat.v1.keras.backend'
52746 ERROR: Hidden import 'tensorflow._api.v2.compat.v1.keras.backend' not found
52746 INFO: Analyzing hidden import 'tensorflow._api.v2.compat.v1.keras.callbacks'
52746 ERROR: Hidden import 'tensorflow._api.v2.compat.v1.keras.callbacks' not found
52746 INFO: Analyzing hidden import 'tensorflow._api.v2.compat.v1.keras.callbacks.experimental'
52746 ERROR: Hidden import 'tensorflow._api.v2.compat.v1.keras.callbacks.experimental' not found
...
52843 INFO: Analyzing hidden import 'tensorflow.compat.v1.config'
52843 ERROR: Hidden import 'tensorflow.compat.v1.config' not found
52843 INFO: Analyzing hidden import 'tensorflow.compat.v1.config.experimental'
52843 ERROR: Hidden import 'tensorflow.compat.v1.config.experimental' not found
...
52995 INFO: Analyzing hidden import 'tensorflow.keras.__internal__'
52995 INFO: Analyzing hidden import 'tensorflow.keras.__internal__.backend'
52995 INFO: Analyzing hidden import 'tensorflow.keras.__internal__.layers'
52995 INFO: Analyzing hidden import 'tensorflow.keras.__internal__.losses'
52995 INFO: Analyzing hidden import 'tensorflow.keras.__internal__.models'
52995 INFO: Analyzing hidden import 'tensorflow.keras.__internal__.utils'
52995 INFO: Analyzing hidden import 'tensorflow.keras.activations'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications.convnext'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications.densenet'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications.efficientnet'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications.efficientnet_v2'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications.imagenet_utils'
52995 INFO: Analyzing hidden import 'tensorflow.keras.applications.inception_resnet_v2'
All the noise in the logs is because --collect-all doesn't do what you think it does. It's an option that'll be getting deleted if I ever get round to it.
I'll try and look at this tomorrow. I don't use tensorflow so can you give me an example sample-app.py to play with and save me having to rummage around online for one.
If the frozen application is working except for being able to find the GPU, perhaps a --copy-metadata tensorflow_metal will help (in addition to already-used switches; perhaps a --copy-metadata tensorflow_macos will also be needed). That's assuming the plugin discovery is done via metadata...
As for the existing tensorflow hook, it is incompatible with tensorflow_macos, because the version checks are made under assumption that the dist name is tensorflow, so we probably mis-identify the version.
@bwoodsend thanks for replying so quickly. I've shared a sample repo showing the issue with simple steps to reproduce and environmental information in the README - hope this makes things easy for you to look into 🙂 Is this enough of an example?
https://github.com/samheather/pyinstaller-issue-demo
@rokm Thanks for the suggestion - yes, I tried adding --copy-metadata for both tensorflow_macos and tensorflow_metal as you suggest but it results in a binary that won't start - so worse off unfortunately.
Looking at the plugin search code in tensorflow/__init__.py, I don't think you'll be able to get it find that libmetal_plugin.dylib unless you patch the tensorflow in your venv.
Around the line 398 in tensorflow/__init__.py, you will find the following code:
# Load all plugin libraries from site-packages/tensorflow-plugins if we are
# running under pip.
# TODO(gunan): Enable setting an environment variable to define arbitrary plugin
# directories.
# TODO(gunan): Find a better location for this code snippet.
from tensorflow.python.framework import load_library as _ll
from tensorflow.python.lib.io import file_io as _fi
# Get sitepackages directories for the python installation.
_site_packages_dirs = []
if _site.ENABLE_USER_SITE and _site.USER_SITE is not None:
_site_packages_dirs += [_site.USER_SITE]
_site_packages_dirs += [_p for _p in _sys.path if 'site-packages' in _p]
if 'getsitepackages' in dir(_site):
_site_packages_dirs += _site.getsitepackages()
if 'sysconfig' in dir(_distutils):
_site_packages_dirs += [_distutils.sysconfig.get_python_lib()]
_site_packages_dirs = list(set(_site_packages_dirs))
# Find the location of this exact file.
_current_file_location = _inspect.getfile(_inspect.currentframe())
def _running_from_pip_package():
return any(
_current_file_location.startswith(dir_) for dir_ in _site_packages_dirs)
if _running_from_pip_package():
# TODO(gunan): Add sanity checks to loaded modules here.
# Load first party dynamic kernels.
_tf_dir = _os.path.dirname(_current_file_location)
_kernel_dir = _os.path.join(_tf_dir, 'core', 'kernels')
if _os.path.exists(_kernel_dir):
_ll.load_library(_kernel_dir)
# Load third party dynamic kernels.
for _s in _site_packages_dirs:
_plugin_dir = _os.path.join(_s, 'tensorflow-plugins')
if _os.path.exists(_plugin_dir):
_ll.load_library(_plugin_dir)
# Load Pluggable Device Library
_ll.load_pluggable_device_library(_plugin_dir)Aside from the fact that PyInstaller does not enable site and that packages' directories do not contain site-packages in their path, the real problem is that _current_file_location = _inspect.getfile(_inspect.currentframe()) yields tensorflow/__init__py; i.e., a relative path due to absolute path stripping in collected code objects. Therefore, _running_from_pip_package() will never evaluate to True in the frozen application, and it won't even attempt to search for the plugins (and even if it did, it would probably fail to find them due to assumptions about path structure).
So you'll need to add an else block to the if _running_from_pip_package(), like this:
if _running_from_pip_package():
# TODO(gunan): Add sanity checks to loaded modules here.
# Load first party dynamic kernels.
_tf_dir = _os.path.dirname(_current_file_location)
_kernel_dir = _os.path.join(_tf_dir, 'core', 'kernels')
if _os.path.exists(_kernel_dir):
_ll.load_library(_kernel_dir)
# Load third party dynamic kernels.
for _s in _site_packages_dirs:
_plugin_dir = _os.path.join(_s, 'tensorflow-plugins')
if _os.path.exists(_plugin_dir):
_ll.load_library(_plugin_dir)
# Load Pluggable Device Library
_ll.load_pluggable_device_library(_plugin_dir)
else: # <--- added code begins here
_tf_dir = _os.path.dirname(__file__)
# Load first party dynamic kernels.
_kernel_dir = _os.path.join(_tf_dir, 'core', 'kernels')
if _os.path.exists(_kernel_dir):
_ll.load_library(_kernel_dir)
_plugin_dir = _os.path.join(_tf_dir, _os.pardir, 'tensorflow-plugins')
if _os.path.exists(_plugin_dir):
_ll.load_library(_plugin_dir)
# Load Pluggable Device Library
_ll.load_pluggable_device_library(_plugin_dir)You need to modify the tensorflow/__init__.py file in your venv (as opposed to the copy collected in your frozen application), and then rebuild the program (pyinstaller sample-app.py --collect-all tensorflow --collect-all tensorflow-plugins --noconfirm --clean).
Trying this now
@rokm I can confirm this works. Thanks a million! I wasn't expecting bespoke support 😀
I'll look at raising an appropriate PR for tensorflow in the coming days to hopefully this doesn't come up again.
I think since they went to all the trouble of making tensorflow-plugins a package, a better approach would be to simply import it (while if it cannot be imported using import statement due to dash, it can be using __import__() or importlib), obtain its __file__ or __path__, and search the corresponding directory. No need for all that site-packages magic, unless there was a very good reason for it.
And unless I'm missing some corner case they are trying to address, _inspect.getfile(_inspect.currentframe()) should be simplified with __file__.