pyinstaller support

Question

pyinstaller support

AlexPaiva opened this issue a year ago · 16 comments

Currently execnet doesn't support pyinstaller, a lot of other packages based on execnet have a lot of raised issues to be supported on pyinstaller, any option/availability to fix this?

Answer 1 · 2023-12-20T14:58:39.000Z

It seems the problem is on how execnet spawns subprocesses: it assumes sys.executable is the Python executable and uses sys.executable -c CODE for execution, but in a frozen pyinstaller application sys.executable points to the main() entry point of the application.

A quick solution I'm just coming up with could be:

execnet needs to "globally" mark that a subprocess is being spawned by execnet. Perhaps a well named environment variable would do the trick.
A pyinstaller runtime hook could look for that "global mark"/environment variable, and then handle the -c command line and execute the code.

This is a rough outline that I think could work.

Full disclosure, I do not have bandwidth to work on this specifically, but wanted to write down this outline in case somebody wants to tackle this.

Answer 2 · 2023-12-20T15:03:03.000Z

It seems the problem is on how execnet spawns subprocesses: it assumes sys.executable is the Python executable and uses sys.executable -c CODE for execution, but in a frozen pyinstaller application sys.executable points to the main() entry point of the application.

A quick solution I'm just coming up with could be:
1. execnet needs to "globally" mark that a subprocess is being spawned by `execnet`. Perhaps a well named environment variable would do the trick.

2. A pyinstaller runtime hook could look for that "global mark"/environment variable, and then handle the `-c` command line and execute the code.
This is a rough outline that I think could work.

Full disclosure, I do not have bandwidth to work on this specifically, but wanted to write down this outline in case somebody wants to tackle this.

Yes that seems like the exact issue, becuse in this case I am using pytest xdist which uses execnet to spawn the processes, and it just starts a new .exe every time I try with pytest xdist when using an .exe with pyinstaller.

I didn't quite understand your solution, can you give me a few more details or a quick example? I can try to tackle this and send a PR

Answer 3 · 2023-12-20T15:20:46.000Z

Sure:

1: before spawning a subprocess via sys.executable, execnet should set the env var EXECNET_SPAWNING_SUBPROCESS=1.
2: a runtime hook is responsible for executing the code inside the frozen executable, something like:

import os, sys

if os.environ.get("EXECNET_SPAWNING_SUBPROCESS") == "1":
    # execnet executes the command line: sys.executable -u -c CODE, so handle that.
    if len(sys.argv) == 4:
        if sys.argv[1:3] == ["-u", "-c"]:
            code = sys.argv[3]
            exec(code)
            sys.exit(0)

In summary, the hook will recognize it is being called by execnet and execute the given Python code, similar to what Python does.

Answer 4 · 2023-12-20T16:03:08.000Z

Sure:

1: before spawning a subprocess via sys.executable, execnet should set the env var EXECNET_SPAWNING_SUBPROCESS=1. 2: a runtime hook is responsible for executing the code inside the frozen executable, something like:
import os, sys

if os.environ.get("EXECNET_SPAWNING_SUBPROCESS") == "1":
    # execnet executes the command line: sys.executable -c CODE, so handle that.
    if len(sys.argv) == 3:
        _, flag, code = sys.argv
        if flag == "-c":
            exec(code)
            sys.exit(0)
In summary, the hook will recognize it is being called by execnet and execute the given Python code, similar to what Python does.

Doesn't seem to work, added that exact code as a run time hook but when calling pytest.main(["-n","2",relative_to_assets('test.py')]) it's still opening a new .exe (-n 2 is to spawn 2 workers which is what pytest-xdist does), something missing? Maybe the sys.argv is not 3?

Answer 5 · 2023-12-20T20:40:35.000Z

Not sure, need to browse execnet's code and see how processes are being spawned... it might be passing some other flags and sys.argv is not 3, as you suggest.

Answer 6 · 2023-12-28T15:10:09.000Z

Not sure, need to browse execnet's code and see how processes are being spawned... it might be passing some other flags and sys.argv is not 3, as you suggest.

After a lot of checking and what not the issue seems to be this: self.popen = p = execmodel.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) at class Popen2IOMaster(Popen2IO): def __init__(self, args, execmodel): in gateway_io.py as it calls python -u -c import sys;exec(eval(sys.stdin.readline())) and since it doesn't recognize the command python as it doesn't have python installed on the target machine since it's a standalone .exe with pyinstaller it doesn't work.

Changing python to sys.executable fixes that but just doesn't work and just opens a new .exe instead.

Any ideas?

Answer 7 · 2023-12-28T15:25:48.000Z

Changing python to sys.executable fixes that but just doesn't work and just opens a new .exe instead.

Does sys.executable point to the exe created by PyInstaller? I think so, in which case the hook I commented before needs to kick in, and handle the -u -c flags (I assumed it was only -c in my example).

EDIT: updated my original example

Answer 8 · 2023-12-28T15:28:23.000Z

Changing python to sys.executable fixes that but just doesn't work and just opens a new .exe instead.

Does sys.executable point to the exe created by PyInstaller? I think so, in which case the hook I commented before needs to kick in, and handle the -u -c flags (I assumed it was only -c in my example).

Yes it points to the .exe, let me check if the hook is properly kicking in

Answer 9 · 2023-12-28T16:07:15.000Z

Changing python to sys.executable fixes that but just doesn't work and just opens a new .exe instead.

Does sys.executable point to the exe created by PyInstaller? I think so, in which case the hook I commented before needs to kick in, and handle the -u -c flags (I assumed it was only -c in my example).

Kicked in with success, and it's running the exec() code properly and the code it is running is: "\n # Source code of gateway_base\n \nexecmodel = get_execmodel('thread')\nio = init_popen_io(execmodel)\nio.write('1'.encode('ascii'))\nserve(io, id='gw0-worker')" but nothing happens, what could be wrong? It's basically doing exec(str(sys.stdin.readline())) and I printed with print(str(str(sys.stdin.readline()))) which returned the above, but it's encased in quotation marks maybe to fit the command line but since it's running on exec() the quotation marks are just running a string so nothing happens? or? With console=True on the pyinstaller spec file I get


================================================= test session starts =================================================
platform win32 -- Python 3.10.5, pytest-7.4.3, pluggy-1.3.0 -- C:\Users\AA\Downloads\Test\main.exe
cachedir: .pytest_cache
rootdir: C:\Users\AA
plugins: xdist-3.5.0
created: 2/2 workers

which seems fine, any ideas?

Answer 10 · 2023-12-28T16:19:46.000Z

It's basically doing exec(str(sys.stdin.readline())) and I printed with print(str(str(sys.stdin.readline())))

Not sure I follow, why are you using sys.stdin.readline() instead of sys.argv[3]? We need to execute code given as argument to the -c parameter.

With console=True on the pyinstaller spec file I get

With sys.stdin.readline() being the code passed to exec? I have no idea what's going on then, we should be executing exec(sys.argv[3]).

Answer 11 · 2023-12-28T16:24:40.000Z

It's basically doing exec(str(sys.stdin.readline())) and I printed with print(str(str(sys.stdin.readline())))

Not sure I follow, why are you using sys.stdin.readline() instead of sys.argv[3]? We need to execute code given as argument to the -c parameter.

With console=True on the pyinstaller spec file I get

With sys.stdin.readline() being the code passed to exec? I have no idea what's going on then, we should be executing exec(sys.argv[3]).

This is the hook except the imports:

if os.environ.get("EXECNET_SPAWNING_SUBPROCESS") == "1":
    # execnet executes the command line: sys.executable -c CODE, so handle that.
    with open('abc.txt', 'w') as file:
        file.write(str(sys.argv)+" || "+str(len(sys.argv)))
    if len(sys.argv) == 4:
        _, flagu, flag, code = sys.argv
        if flag == "-c":
            print(str(code))
            exec(code)
            sys.exit(0)

the print returns import sys;exec(eval(sys.stdin.readline())) which is what is in the code variable.
Nothing was happening so I also added this print for further debugging:
print(str(sys.stdin.readline())) which is basically what the exec() is going to do so I could see what was being sent, this returned "\n # Source code of gateway_base\n \nexecmodel = get_execmodel('thread')\nio = init_popen_io(execmodel)\nio.write('1'.encode('ascii'))\nserve(io, id='gw0-worker')"

Answer 12 · 2023-12-28T16:46:34.000Z

Ahh OK sorry, now I get what you mean by import sys;exec(eval(sys.stdin.readline())).

Hmm this is what we do at ESSS to support this:

    if "-c" in sys.argv:
        i = sys.argv.index("-c")
        if i + 1 < len(sys.argv):
            cmd = sys.argv[i + 1]
            sys.argv.remove("-c")
            sys.argv.remove(cmd)
            exec(cmd, {})
            sys.exit(0)

So we also take care of removing -c and the code from sys.argv to make sure it does not affect the executing code. With this we can get pytest-xdist to work (not sure if you are using it for the same purpose, or something else).

Answer 13 · 2023-12-28T16:56:20.000Z

Just want to use pytest-xdist to run two instances of the same test isolated, it works perfectly with command line but wanted it to work with pyinstaller aswell.

Changed the hook to:

if os.environ.get("EXECNET_SPAWNING_SUBPROCESS") == "1":
    # execnet executes the command line: sys.executable -c CODE, so handle that.
    with open('abc.txt', 'w') as file:
        file.write(str(sys.argv)+" || "+str(len(sys.argv)))
    if len(sys.argv) == 4:
        _, flagu, flag, code = sys.argv
        if "-c" in sys.argv:
            i = sys.argv.index("-c")
            if i + 1 < len(sys.argv):
                cmd = sys.argv[i + 1]
                sys.argv.remove("-c")
                sys.argv.remove(cmd)
                exec(cmd, {})
                sys.exit(0)

Ran it, got this error:

================================================= test session starts =================================================
platform win32 -- Python 3.10.5, pytest-7.4.3, pluggy-1.3.0 -- C:\Users\AA\Downloads\Test\main.exe
cachedir: .pytest_cache
rootdir: C:\Users\AA
plugins: xdist-3.5.0
created: 2/2 workersTraceback (most recent call last):
  File "runtime_hook.py", line 14, in <module>
    exec(cmd, {})
  File "<string>", line 1, in <module>
  File "<string>", line 4, in <module>
NameError: name 'get_execmodel' is not defined
[3036] Failed to execute script 'runtime_hook' due to unhandled exception!

Seems that it's now properly executing the exec() and as seen in the string I sent 1 reply above which is the code that is ran it's giving error here execmodel = get_execmodel('thread'), missing some imports maybe or?

Answer 14 · 2023-12-28T17:36:39.000Z

Hmm yeah seems like it is missing some imports, or assuming it is executing in some kind of environment... but at this point I'm not sure how to proceed I'm afraid.

Answer 15 · 2023-12-28T19:13:08.000Z

Hmm yeah seems like it is missing some imports, or assuming it is executing in some kind of environment... but at this point I'm not sure how to proceed I'm afraid.

By adding

import_statements = """
import os, sys
sys.path.append(os.path.join(sys._MEIPASS, "execnet"))
sys.path.append(os.path.join(sys._MEIPASS))
import execnet
from execnet.gateway_base import *
"""

and

if "-c" in sys.argv:
            i = sys.argv.index("-c")
            if i + 1 < len(sys.argv):
                cmd = sys.argv[i + 1]
                sys.argv.remove("-c")
                sys.argv.remove(cmd)
                cmd = import_statements + cmd
                exec(cmd, {})
                sys.exit(0)

It moved onwards, Now it's giving this error:

initialized: 1/2 workersINTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "_pytest\main.py", line 269, in wrap_session
INTERNALERROR>   File "pluggy\_hooks.py", line 493, in __call__
INTERNALERROR>   File "pluggy\_manager.py", line 115, in _hookexec
INTERNALERROR>   File "pluggy\_callers.py", line 152, in _multicall
INTERNALERROR>   File "pluggy\_result.py", line 114, in get_result
INTERNALERROR>   File "pluggy\_callers.py", line 77, in _multicall
INTERNALERROR>   File "xdist\dsession.py", line 83, in pytest_sessionstart
INTERNALERROR>   File "xdist\workermanage.py", line 68, in setup_nodes
INTERNALERROR>   File "xdist\workermanage.py", line 68, in <listcomp>
INTERNALERROR>   File "xdist\workermanage.py", line 76, in setup_node
INTERNALERROR>   File "xdist\workermanage.py", line 277, in setup
INTERNALERROR>   File "execnet\gateway.py", line 121, in remote_exec
INTERNALERROR>     source = inspect.getsource(source)
INTERNALERROR>   File "inspect.py", line 1147, in getsource
INTERNALERROR>   File "inspect.py", line 1129, in getsourcelines
INTERNALERROR>   File "inspect.py", line 958, in findsource
INTERNALERROR> OSError: could not get source code

It's looking for the xdist\remote.pyc file, any idea where I can find it? to manually import it as data to pyinstaller

Answer 16 · 2023-12-28T19:28:36.000Z

Just fixed it and it works! Had to re-import the pytest-xdist as xdist aswell as data on the .spec file for it to work.
Working perfectly! Thanks for your help @nicoddemus

For it to work you need the hook we discussed added at runtime, along with the duplicated import of pytest-xdist as data with pytest-xdist and xdist as the names, and some slight modifications to the source code of execnet gateway_io.py and gateway.py will try to clean the code up and submit as a PR.

Hooks used:
hood-pytest_xdist.py

from PyInstaller.utils.hooks import copy_metadata, collect_submodules

# For pytest_xdist
datas = copy_metadata('pytest_xdist')
hiddenimports = collect_submodules('pytest_xdist')

# Additionally, for execnet
datas += copy_metadata('execnet')
hiddenimports += collect_submodules('execnet')

runtime_hook.py

import os, sys
sys.path.append(os.path.join(sys._MEIPASS, "execnet"))
sys.path.append(os.path.join(sys._MEIPASS))
import execnet
from execnet.gateway_base import *
import_statements = """
import os, sys
sys.path.append(os.path.join(sys._MEIPASS, "execnet"))
sys.path.append(os.path.join(sys._MEIPASS))
import execnet
from execnet.gateway_base import *
"""
if os.environ.get("EXECNET_SPAWNING_SUBPROCESS") == "1":
    if len(sys.argv) == 4:
        _, flagu, flag, code = sys.argv
        if "-c" in sys.argv:
            i = sys.argv.index("-c")
            if i + 1 < len(sys.argv):
                cmd = sys.argv[i + 1]
                sys.argv.remove("-c")
                sys.argv.remove(cmd)
                cmd = import_statements + cmd
                exec(cmd, {})
                sys.exit(0)

On the pyinstaller .spec file datas=[('./pytest_xdist', 'xdist'),('./pytest_xdist', 'pytest_xdist'),('...Your own python library directory....AppData/Local/Programs/Python/Python310/Lib/site-packages/execnet', 'execnet'), .... any others .... ], hookspath=['hooks'], and runtime_hooks=['runtime_hook.py'], (I added the runtime_hook.py to the hooks folder and to the directory of the spec file. Also on hiddenimports add pytest, pytest xdist and execnet just as a backup. Again, dirty but works. Once I get it clean I will send a PR.

You need to also perform some source code modifications which I will send on the PR and tag this issue.

Again, thanks for your help @nicoddemus