[Bug]: Calling a py_binary in a subprocess doesn't work if env flags are passed

[mvukov]

What happened?

Calling a py_binary in a subprocess fails if called with extra env vars. I prepared a demonstration in this branch: https://github.com/oqton/rules_py/tree/feature/subprocess_env_bug.

Here is the output I get:

bazelisk test //py/tests/binary_calls_binary/second
INFO: Analyzed target //py/tests/binary_calls_binary/second:second (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //py/tests/binary_calls_binary/second:second (see /home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/execroot/aspect_rules_py/bazel-out/k8-fastbuild/testlogs/py/tests/binary_calls_binary/second/second/test.log)
INFO: From Testing //py/tests/binary_calls_binary/second:second:
==================== Test output for //py/tests/binary_calls_binary/second:second:
ERROR: cannot find bazel_tools/tools/bash/runfiles/runfiles.bash
Traceback (most recent call last):
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/sandbox/linux-sandbox/27/execroot/aspect_rules_py/bazel-out/k8-fastbuild/bin/py/tests/binary_calls_binary/second/second.runfiles/aspect_rules_py/py/tests/binary_calls_binary/second/second_main.py", line 3, in <module>
    subprocess.check_call('py/tests/binary_calls_binary/first/first', env={'FOO': 'bar'})
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/execroot/aspect_rules_py/external/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'py/tests/binary_calls_binary/first/first' returned non-zero exit status 1.
================================================================================
Target //py/tests/binary_calls_binary/second:second up-to-date:
  bazel-bin/py/tests/binary_calls_binary/second/second
INFO: Elapsed time: 1.692s, Critical Path: 1.27s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//py/tests/binary_calls_binary/second:second                             FAILED in 0.6s
  /home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/execroot/aspect_rules_py/bazel-out/k8-fastbuild/testlogs/py/tests/binary_calls_binary/second/second/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

Without env flags in the subprocess the test succeeds.

Version

Development (host) and target OS/architectures:

Output of bazel --version: 5.3.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

rules_py: based on c89d948.

Language(s) and/or frameworks involved:

Python

How to reproduce

# Call this when the aforementioned branch is checked out.
bazelisk test //py/tests/binary_calls_binary/second

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

[mvukov]

It turns out, the error is on my end, per documentation of subprocess.run. If env is specified it overrides the caller process env. So, something like to following fixes the test:

import os
import subprocess

env = os.environ
env.update({'FOO': 'bar'})
subprocess.check_call('py/tests/binary_calls_binary/first/first', env=env)

The old, non-working version, was:

import subprocess

subprocess.check_call('py/tests/binary_calls_binary/first/first', env={'FOO': 'bar'})

Nevertheless, its strange that I get ERROR: cannot find bazel_tools/tools/bash/runfiles/runfiles.bash error as the pwd essentially stays the same when the second app calls the first app in my example.

[mattem]

I'm not clear if there's work for us to do here, or this can be closed?

[mvukov]

Well, at first the behavior of rules_py in this case was unexpected (works with rules_python :) ). Then I learned that my code was incomplete. Once I fixed it, then everything is fine. Let's close this.