Multiprocessing Manager Class Causes Infinite Loop on Mac OS (intel)
IsaiahCoroama opened this issue · 18 comments
Bug Description
When I try to create a multiprocessing.Manager()
object, even with freeze_support
, it causes an infinite loop.
Steps To Reproduce
Create a main.py
file.
is_darwin = True # This is determined by a module that does not affect the results.
def main():
...
if __name__ == "__main__":
if is_darwin:
import multiprocessing; multiprocessing.freeze_support()
print("Creating Managers")
manager_1 = multiprocessing.Manager()
print("Finished Creating Manager #1")
manager_2 = multiprocessing.Manager()
print("Finished Creating Manager #2")
manager_3 = multiprocessing.Manager()
print("Finished Creating Manager #3")
print("Finished Creating ALL Managers.")
main()
Create a setup.py
file.
from cx_Freeze import setup, Executable
build_exe_options = {
["zip_include_packages"]: []
}
setup(
name="main-demo",
version="0.0.0",
options={"build_exe": build_exe_options},
executables=[Executable("main.py", base="console")]
)
Run the setup file with the following configuration:
python setup.py bdist_mac
Run the generated application file:
./build/main-demo-1.0.0.app/Contents/MacOS/main
Expected behavior
From the demo, it prints "Creating Managers" repeatedly indefinity.
System Information:
- Platform information:
Mac OS (Sonoma 14.4)
- OS architecture:
x86_64
- cx_Freeze version:
7.1.0.post0
- Python version:
3.12.3
Additional context
Within the project, I am using managers to share data between three separate processes. These processes run a loop attached to a manager.Event()
object. In the loop, it checks if a boolean value within a manager.dict()
object is True. If it is, it will call a tkinter
function; these consist of filedialog.askdirectory()
, messagebox.showerror()
, and simpledialog.askstring()
. Then, it sets the output to a value within the manager dictionary. The process is then repeated with a few other steps. If you would like, I can post the full source code, but currently, the issue only remains with creating the first manager object. I have gotten this to work with PyInstaller
, but I need to create a universal2
application, and cx_Freeze
has been the closest I have gotten to finally completing everything I need.
Although it states in the official documentation that multiprocessing.freeze_support()
only works on Windows, it has fixed this same infinite loop within PyInstaller
when I was trying to get that to work.
Disclaimer
If there are any misspellings within the code above, that is an accident and not in my actual code. I wrote this issue from my Windows machine all by hand.
After looking into the source code of the hook function for the multiprocessing module, I read the comment about multiprocessing.spawn.freeze_support()
working on all operating systems and tried running it as well. It fixed the issue. I'm sorry I didn't figure this out sooner and made an issue for it when there was already a fix. I at least hope that this issue helps others who may run into this problem.
is_darwin = True # This is determined by a module that does not affect the results.
def main():
...
if __name__ == "__main__":
if is_darwin:
import multiprocessing.spawn
import multiprocessing
multiprocessing.spawn.freeze_support() # This fixes the issue.
print("Creating Managers")
manager_1 = multiprocessing.Manager()
print("Finished Creating Manager #1")
manager_2 = multiprocessing.Manager()
print("Finished Creating Manager #2")
manager_3 = multiprocessing.Manager()
print("Finished Creating Manager #3")
print("Finished Creating ALL Managers.")
main()
I use 3 tests on Windows, Linux, and macOS.
Does this test work for you?
https://github.com/marcelotduarte/cx_Freeze/blob/main/tests/test_multiprocessing.py#L53-L63
https://github.com/marcelotduarte/cx_Freeze/actions/runs/9274051105/job/25515543065#step:6:297
I reviewed all of your test cases with the following setup.
[sample1.py]
import multiprocessing, sys
def foo(q):
q.put('hello')
if __name__ == '__main__':
if sys.platform == 'win32': # the conditional is unecessary
multiprocessing.freeze_support()
multiprocessing.set_start_method('spawn')
q = multiprocessing.SimpleQueue()
p = multiprocessing.Process(target=foo, args=(q,))
p.start()
print(q.get())
p.join()
[sample2.py]
import multiprocessing, sys
def foo(q):
q.put('hello')
if __name__ == '__main__':
ctx = multiprocessing.get_context('spawn')
if sys.platform == 'win32': # the conditional is unecessary
ctx.freeze_support()
q = ctx.Queue()
p = ctx.Process(target=foo, args=(q,))
p.start()
print(q.get())
p.join()
[sample3.py]
if __name__ == "__main__":
import multiprocessing, sys
if sys.platform == 'win32': # the conditional is unecessary
multiprocessing.freeze_support()
multiprocessing.set_start_method('spawn')
mgr = multiprocessing.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
[setup.py]
from cx_Freeze import Executable, setup
setup(
name="test_multiprocessing",
version="0.1",
description="Sample for test with cx_Freeze",
executables=[
Executable("sample1.py"),
Executable("sample2.py"),
Executable("sample3.py"),
],
options={
"build_exe": {
"excludes": ["tkinter"],
"silent": True,
}
}
)
[sample1.py Run Command]
./build/exe.macos-10.9-universal2-3.12/sample1
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample1
[sample2.py Run Command]
./build/exe.macos-10.9-universal2-3.12/sample2
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample2
[sample3.py Run Command]
./build/exe.macos-10.9-universal2-3.12/sample3
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
[setup.py Run Command]
python setup.py bdist_mac
Output
From the following tests above, all produced the expected output. For sample 3, I created an altered test case by removing the multiprocessing.set_start_method("spawn")
and was able to reproduce the infinite spawn recursion as before.
Alternate Test Case For Sample 3
if __name__ == "__main__":
import multiprocessing, sys
if sys.platform == 'win32': # the conditional is unecessary
multiprocessing.freeze_support()
# multiprocessing.set_start_method('spawn') Removed for testing purposes
print("Creating Manager")
mgr = multiprocessing.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
[Run Command]
./build/exe.macos-10.9-universal2-3.12/sample3
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
[Output]
user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample3
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...
Conclusion
From the following tests, not calling either mutliprocessing.spawn.freeze_support()
or multiprocessing.set_start_method("spawn")
will cause the multiprocessing.Manager()
to run in an infinite loop.
Your conclusion is interesting, because according to the Python documentation, spawn is the default start method. Then there would be no need to configure it, and in the set_start_method documentation it states "that this should be called at most once".
However based on the examples, I think that the alternative (correct?) for sample3 is:
if __name__ == "__main__":
import multiprocessing
mp_context = multiprocessing.get_context('spawn')
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
Also, in the The spawn and forkserver start methods section has "Safe importing of main module" with recommendations to use set_start_method.
Can you check if the default start method is indeed spawn in python and in the executable?
freeze_support
from spawn
worked for my full use case but freeze_support
from multiprocessing directly did not. How odd. I didn't try the trimmed down example, but my work is with Queue and Process
[sample4.py]
if __name__ == "__main__":
import multiprocessing
mp_context = multiprocessing.get_context('spawn')
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
Output
user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample3
Creating Manager
creating dict...done!
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
Creating Manager
creating dict...done!
Alternate Test Case For Sample 4
if __name__ == "__main__":
import multiprocessing
#mp_context = multiprocessing.get_context('spawn') Removed for test purposes
print("Creating Manager")
mgr = multiprocessing.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
Output of Alternate Test Case
user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...
The Default Start Method
if __name__ == "__main__":
import multiprocessing
print(multiprocessing.get_start_method()) # The default start method
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
Output
user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
spawn
Creating Manager
spawn
Creating Manager
spawn
Creating Manager
...
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
spawn
Creating Manager
spawn
Creating Manager
spawn
Creating Manager
...
Another Test Case with Getting the Default Spawn Method
if __name__ == "__main__":
import multiprocessing.spawn
import multiprocessing
multiprocessing.spawn.freeze_support()
print(multiprocessing.get_start_method()) # The default start method
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")
Output
user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
spawn
Creating Manager
creating dict...done!
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
spawn
Creating Manager
creating dict...done!
Off Topic Question
I've noticed that "universal2" is labeled in the name of the generated build_exe
directory, but it is not a standalone file. Is there a way to create standalone universal2
executable files using cx_Freeze
?
Trimmed example for context:
from multiprocessing import Process, freeze_support
def f():
print("Hello from cx_Freeze")
if __name__ == "__main__":
freeze_support()
Process(target=f).start()
The default in python is spawn
and it works if you set it to fork
. However, the notice in the FAQ could probably just be updated to mention both freeze support functions without any additional changes
@IsaiahCoroama read this.
For "Alternate Test Case For Sample 4" the mgr = mp_context.Manager()
was changed to mgr = multiprocessing.Manager()
to run?
@ntindle I should change the FAQ or redirect freeze_support in the hook to avoid future questions....
My naive understanding is that you sometimes need one vs the other depending on if you want to spawn or fork which differs by platform
@IsaiahCoroama read this. For "Alternate Test Case For Sample 4" the
mgr = mp_context.Manager()
was changed tomgr = multiprocessing.Manager()
to run?@ntindle I should change the FAQ or redirect freeze_support in the hook to avoid future questions....
Apologies for the confusion. You're right, it should be multiprocessing.Manager()
instead of mp_context.Manager()
. I'll update the post above with the correct text. Additionally, I've rerun the test, and it yielded the same output as previously mentioned.
I've reviewed the information you provided, and I'm utilizing the universal2
version of Python. However, upon running the file
command on the generated executables, I noticed they only show the x86_64
architecture. Are you suggesting that the universal2
part of the filename is merely for display purposes and doesn't reflect the actual build generation? Or am I misunderstanding the information you provided?
I don't use macOS, but I know that other people use universal2, and I recently stopped generating universal2 and started generating it again. But from what I understand, the python and wheel packages installed must be universal2 to work.
Believe it or not, I don't use Mac OS either; I'm only borrowing a Mac computer because I'm attempting to develop a multi-platform product. I ran the file
command on a Python base library like _ctypes.cpython-312-darwin.so
, and it displayed support for both x86_64
and arm64
, indicating that the built-in libraries are indeed universal2
. However, I created a simple script that walks through the exe.macosx-10.9-universal2-3.12
build directory and checks all files architecture. From it, I found that every single library file, files with the .so
file extension, were universal2
but the main executable files generated by cx_Freeze
were only the native architecture, in my case x86_64
.
The File I Used to Check the Architectures of the Generated Binary Files
import subprocess
import os
def check_arch(filename):
output = subprocess.check_output(["file", filename]).decode()
if "x86_64" in output and "arm64" in output:
arch = "universal2"
elif "x86_64" in output:
arch = "x86_64"
elif "arm64" in output:
arch = "arm64"
elif "data" in output: # Typically files with the (.pyc) just output data by the file command
arch = "data"
else:
arch = "unknown"
arch = f"[{arch}]:" # Formatting
print(f"{arch:<14}{filename}")
def main():
path = os.path.abspath("multiprocessing/build/exe-macosx-10.9-universal2-3.12")
for root, _, files in os.walk(path):
for file in files:
check_arch(os.path.join(root, file))
For the output, I cut down the path length to make it easier to read.
Output
Console Output
[x86_64]: /exe.macosx-10.9-universal2-3.12/sample2 [x86_64]: /exe.macosx-10.9-universal2-3.12/sample3 [x86_64]: /exe.macosx-10.9-universal2-3.12/sample4 [unknown]: /exe.macosx-10.9-universal2-3.12/frozen_application_license.txt [x86_64]: /exe.macosx-10.9-universal2-3.12/sample1 [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_struct.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/array.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_queue.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_blake2.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha1.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixshmem.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_hashlib.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/grp.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_socket.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_decimal.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_csv.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_statistics.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha3.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/fcntl.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/Python [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ssl.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/zlib.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/unicodedata.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bisect.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_cn.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multibytecodec.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/libssl.3.dylib [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixsubprocess.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/libcrypto.3.dylib [unknown]: /exe.macosx-10.9-universal2-3.12/lib/library.dat [universal2]: /exe.macosx-10.9-universal2-3.12/lib/select.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_lzma.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/pyexpat.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_pickle.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_jp.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bz2.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ctypes.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/mmap.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_datetime.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/math.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha2.cpython-312-darwin.so [data]: /exe.macosx-10.9-universal2-3.12/lib/library.zip [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_iso2022.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_tw.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_opcode.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_kr.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_heapq.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_md5.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multiprocessing.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_random.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_contextvars.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_hk.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/binascii.cpython-312-darwin.so [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp861.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp875.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/quopri_codec.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp874.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp860.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_cyrillic.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1140.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp862.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp863.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_32.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_greek.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_8.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp720.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/oem.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp866.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_latin2.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_8.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1026.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp737.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp858.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp864.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp865.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_9.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1256.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp037.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_kr.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/shift_jis.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/big5.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/johab.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/zlib_codec.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1257.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1255.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_16_le.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/gb2312.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp949.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/koi8_t.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp424.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_32_le.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/koi8_u.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/hz.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1254.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1250.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/rot_13.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/kz1048.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/unicode_escape.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_ext.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1251.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/base64_codec.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1253.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/uu_codec.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1125.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/palmos.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/koi8_r.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp437.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1252.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/ascii.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/tis_620.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_32_be.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_roman.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/big5hkscs.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/undefined.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/idna.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_16_be.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/latin_1.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_2004.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/aliases.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_croatian.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/gbk.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/ptcp154.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/raw_unicode_escape.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp950.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_arabic.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1258.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_romanian.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp775.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_iceland.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/shift_jisx0213.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/hp_roman8.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_jis_2004.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_4.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_7.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/charmap.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_turkish.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/hex_codec.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/bz2_codec.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_16.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp869.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp855.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/gb18030.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_5.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_7.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp857.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_14.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_kr.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_15.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/punycode.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mbcs.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp856.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_8_sig.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_6.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_2.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp932.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_1.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_11.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp852.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_jisx0213.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/shift_jis_2004.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_10.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_16.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_3.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_1.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp500.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_2.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_jp.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_farsi.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp850.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_13.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_3.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1006.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/encodings/cp273.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/ctypes/_endian.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/ctypes/__init__.pyc [unknown]: /exe.macosx-10.9-universal2-3.12/lib/ctypes/macholib/fetch_macholib [unknown]: /exe.macosx-10.9-universal2-3.12/lib/ctypes/macholib/README.ctypes [unknown]: /exe.macosx-10.9-universal2-3.12/lib/ctypes/macholib/fetch_macholib.bat [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/sharedctypes.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/connection.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/resource_tracker.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/pool.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_fork.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/heap.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/synchronize.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/context.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/reduction.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/resource_sharer.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/forkserver.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/queues.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_forkserver.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/process.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/shared_memory.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/managers.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/util.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/spawn.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_spawn_posix.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_spawn_win32.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/dummy/connection.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/dummy/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/urllib/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/urllib/parse.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/zipfile/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/zipfile/_path/glob.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/zipfile/_path/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/xml/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/xml/parsers/expat.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/xml/parsers/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/http/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/http/client.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/_bootstrap.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/abc.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/_bootstrap_external.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/simple.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/_abc.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/readers.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/machinery.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/util.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_adapters.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/abc.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/simple.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/readers.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_legacy.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_common.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_itertools.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_adapters.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_meta.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_collections.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_text.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_functools.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_itertools.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/xmlrpc/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/xmlrpc/client.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/collections/abc.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/collections/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/logging/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/encoders.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/parser.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/policy.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/_policybase.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/charset.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/_parseaddr.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/errors.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/iterators.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/contentmanager.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/base64mime.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/headerregistry.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/_header_value_parser.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/header.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/quoprimime.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/feedparser.pyc [unknown]: /exe.macosx-10.9-universal2-3.12/lib/email/architecture.rst [data]: /exe.macosx-10.9-universal2-3.12/lib/email/message.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/generator.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/utils.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/email/_encoded_words.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/re/_parser.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/re/_compiler.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/re/_casefix.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/re/__init__.pyc [data]: /exe.macosx-10.9-universal2-3.12/lib/re/_constants.pyc
Same Test Code with .so
Filter
import subprocess
import os
def check_arch(filename):
output = subprocess.check_output(["file", filename]).decode()
if "x86_64" in output and "arm64" in output:
arch = "universal2"
elif "x86_64" in output:
arch = "x86_64"
elif "arm64" in output:
arch = "arm64"
elif "data" in output: # Typically files with the (.pyc) just output data by the file command
arch = "data"
else:
arch = "unknown"
arch = f"[{arch}]:" # Formatting
print(f"{arch:<14}{filename}")
def main():
path = os.path.abspath("multiprocessing/build/exe-macosx-10.9-universal2-3.12")
for root, _, files in os.walk(path):
for file in files:
if not file.endswith(".so"): # Skip non-library files
continue
check_arch(os.path.join(root, file))
Output
Console Output
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_struct.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/array.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_queue.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_blake2.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha1.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixshmem.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_hashlib.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/grp.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_socket.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_decimal.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_csv.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_statistics.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha3.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/fcntl.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ssl.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/zlib.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/unicodedata.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bisect.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_cn.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multibytecodec.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixsubprocess.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/select.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_lzma.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/pyexpat.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_pickle.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_jp.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bz2.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ctypes.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/mmap.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_datetime.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/math.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha2.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_iso2022.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_tw.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_opcode.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_kr.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_heapq.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_md5.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multiprocessing.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_random.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_contextvars.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_hk.cpython-312-darwin.so [universal2]: /exe.macosx-10.9-universal2-3.12/lib/binascii.cpython-312-darwin.so
Executable File Architectures
[x86_64]: /exe.macosx-10.9-universal2-3.12/sample2
[x86_64]: /exe.macosx-10.9-universal2-3.12/sample3
[x86_64]: /exe.macosx-10.9-universal2-3.12/sample4
[x86_64]: /exe.macosx-10.9-universal2-3.12/sample1
Conclusion
Based on the data provided, it appears that even when utilizing universal2
binary files obtained from installing the universal2
version of Python, cx_Freeze
does not generate corresponding universal2
binary files. This suggests that there might be limitations or configurations within cx_Freeze
's executable generation that prevents the creation of universal2
binaries alongside it.
Uninstall cx_Freeze and reinstall to check the wheel installed, or download https://files.pythonhosted.org/packages/e8/12/70285b18f9dea354e69ee1b472f46e79f9ae9aa6749d23a3109bb73aa9fc/cx_Freeze-7.1.0.post0-cp312-cp312-macosx_10_9_universal2.whl and install it, so you should get a universal2 executable.
Based on information from you and others, I improved the hook for multiprocessing.
You can test the patch in the latest development build:
pip install --force --no-cache --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze
The provisional documentation: https://cx-freeze--2443.org.readthedocs.build/en/2443/faq.html#multiprocessing-support
Release 7.1.1 is out!
Documentation