Use stdin in child process

[pawamoy]

Is it possible?

I'd like to spawn a process that reads stdin.

cat requirements.txt | ./script_that_spawns.py safety --check --stdin

When doing so and trying to read safety output, it blocks, and I have to interrupt it with control-c:

Traceback (most recent call last):
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/bin/failprint", line 8, in <module>
    sys.exit(main())
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 125, in main
    return run(
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 54, in run
    output.append(process.read())
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 818, in read
    b = super(PtyProcessUnicode, self).read(size)
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 516, in read
    s = self.fileobj.read1(size)
KeyboardInterrupt

Here is the actual Python code I'm using:

process = PtyProcessUnicode.spawn(cmd)

output = []

while True:
    try:
        output.append(process.read())
    except EOFError:
        break

process.close()

[takluyver]

As things stand, to do this with ptyprocess your spawn wrapper would need to read its own stdin, and write the data to the terminal of its child process.

But in most cases, it shouldn't be necessary to use a pty, so you can use subprocess.Popen(..., stdin=sys.stdin, stdout=subprocess.PIPE) to pass stdin through.

[pawamoy]

The pty is required to keep the colors (actually, prevent the program to modify its output) 😄

But yeah meanwhile I have implemented a --no-pty option to use subprocess.Popen instead.

Would this be relatively easy to implement in ptyprocess? I'm not particularly comfortable with such things but I could try and send a PR if you'd be willing to review it.

[takluyver]

Some programs have options to produce coloured output even when their output is piped, e.g. ls --color=always . If the one you're using doesn't, one possibility would be to see if they're interested in adding it (either a command line option or an environment variable). Detecting 'am I in a terminal' makes sense as the default, but it's also useful at times to have a way to override it.

To be honest, I'm not excited about the idea of adding the feature you mention to ptyprocess. It's not particularly a bad idea - at least I haven't thought of any obvious problem. It's just that I don't use this project much nowadays, and even good changes that someone else makes a PR for always mean work - reviewing code and docs, making releases, responding to the inevitable issues when it doesn't work in some corner case, or it's not quite what people think it is...

Feel free to make the PR anyway, if you want. There are other maintainers on the project, and maybe someone will be more interested than me. But I don't want to mislead you - it's not a very active codebase.

[pawamoy]

OK, thank you for being straightforward, I appreciate it 🙂

Some programs have options to produce coloured output even when their output is piped, e.g. ls --color=always

This is the ideal world indeed. The --color=<auto|never|always> gives control to the end-user. I myself try to always implement it in my projects when they use colors.

Unfortunately the tool I'm building is meant to run arbitrary commands, so I don't see myself going everywhere opening issues for adding the --color option hehe.

Anyway, I don't think I'll send a PR here as this is far from a being something I critically need, and I agree that introducing new code would very likely require maintenance.
I'll wait to see if there's more feedback from other maintainers, and reconsider later. Thanks again!

[pawamoy]

I was eventually able to feed input to the subprocess, but noticed two issues.

def run_pty_subprocess(cmd, stdin=None):
    process = PtyProcessUnicode.spawn(cmd)
    pty_output: List[str] = []

    if stdin is not None:
        process.write(stdin)
        process.sendeof()

    while True:
        try:
            output_data = process.read()
        except EOFError:
            break
        pty_output.append(output_data)

    process.close()
    output = "".join(pty_output)

    # Issue 1: The text I send to the process (with `write`) appears again in the process output.
    # I have to remove it from the output.
    if stdin is not None:
        if output.startswith(stdin):
            output = output[len(stdin):]
        else:
            # Issue 2: Carriage returns are inserted in the output!
            # Is this expected? Note that I'm *not* on Windows.
            stdin_cr = stdin.replace("\n", "\r\n")
            if output.startswith(stdin_cr):
                output = output[len(stdin_cr):]

    code = process.exitstatus
    return code, output

[carlmontanari]

@pawamoy -- Looks like you've already got it under control, but am watching this repo/am a grateful user of it and saw and figured I could chime in!

afaik, both of those issues are expected/normal (at least on *nix systems). I built a ssh client around ptyprocess and do similar to what you've posted here -- every time I write something I just read up through that so that it gets "consumed" off the buffer. Also as part of my read method I just sub out \r with nothing.

Probably not apples to apples and may be more work than its worth, but you may be able to steal some stuff from here -- this section is probably the most relevant bits.

[Red-M]

iirc, you should be able to turn echo off in the env when feeding your data in to turn off the same output being spat back to you when you get the output of the 2nd command.

@carlmontanari I may have to provide you another transport for that library ;)

[carlmontanari]

@Red-M 😁 (I'm guessing redlibssh2?) I would be very open to that! ssh2-python is already in scrapli but it doesn't look like its getting tons of maintenance love, so seems unclear if/when 3.10 wheels will exist! Guessing you had the same concerns and hence your fork! In any case, very open to possibility of swapping in redlibssh2 -- I guess feel free to open an issue/discussion/pr or feel free to email me at carl dot montanari at gmail

[pawamoy]

Hey @carlmontanari, @Red-M, so nice to see other devs chiming in 😄

you should be able to turn echo off

Oh, why did I not think of this. I'll try it and report back, thanks!

I'm also experimenting with the pty module directly, I'll post something here if it works correctly on both Linux and Windows!

[pawamoy]

Well of course it does not work on Windows. I'll stick to ptyprocess and turn echo off in the child process 🙂

[pawamoy]

OK, final piece of code. I had to send EOF twice so the read would not block forever. Not sure why.

def run_pty_subprocess(cmd, stdin=None):
    process = PtyProcessUnicode.spawn(cmd)
    pty_output: List[str] = []

    if stdin is not None:
        process.setecho(False)
        process.write(stdin)
        process.sendeof()
        # not sure why but sending only one eof is not enough,
        # and it blocks forever at process.read()
        process.sendeof()

    while True:
        try:
            output_data = process.read()
        except EOFError:
            break
        pty_output.append(output_data)

    process.close()
    output = "".join(pty_output).replace("\r\n", "\n")
    return process.exitstatus, output

[takluyver]

This SO answer might be relevant for why a single EOF doesn't suffice. .sendeof() is equivalent to Ctrl-D interactively.