Pass-through of a Program's Standard Error

[jesselang]

Hello, and thank you for this wonderful package!

I'd like to use script with some programs that use standard error to prompt the user to do something before providing output. One example is ykman which prompts the user to "Touch your YubiKey...". After doing so, the program will print the one time password code to standard out. Ideally, I'd like the program's standard error to pass-through directly to os.Stderr, but it appears that script.Exec() instead interleaves the program's stderr and stdout. This means the user of this script doesn't receive the prompt from the program, and also means the prompt needs to be filtered out of the output. I'm curious if anyone has come up with a way to handle this. Below is a minimal example. Thanks!

package main

import (
	"github.com/bitfield/script"
)

func main() {
	p := script.Exec(`
		ykman oath accounts code --single "someAccountId"
	`)
	code, _ := p.String()
	script.Echo(code).Stdout()
}

[bitfield]

Bearing in mind that Exec commands run concurrently, I think you should see the output as it's printed. Try something like this, for example:

cmdLine := `ykman oath accounts code --single "someAccountId"`
script.Exec(cmdLine).Stdout()

You should see the prompt right away, and then once you've mashed the device, it should print the code. Does that work?

[jesselang]

You should see the prompt right away, and then once you've mashed the device, it should print the code. Does that work?

Thanks @bitfield. Your example does work, but in my case, I want to capture standard output (the code) as a string to pass to another pipe later in the script, while passing along standard error (preferably to os.Stderr, but os.Stdout would suffice) to provide the user with prompts and any error text. I'm brand new to script, so I'm probably just missing some understanding of how I can do that. Any further ideas for this use case? Thanks!

[jesselang]

A more complete example demonstrating what I'm doing with the code.

package main

import (
	"errors"
	"fmt"
	"os"
	"os/user"

	"github.com/bitfield/script"
)

func main() {
	p := script.Exec(`
		ykman oath accounts code --single "someAccountId"
	`)
	out, _ := p.Reject("Touch your YubiKey...").String()

	script.Exec(fmt.Sprintf(`
		aws-vault exec --mfa-token=%s profile --
			aws sts get-caller-identity
	`, out)).Stdout()
}

[bitfield]

You can insert any arbitrary function in-line in a pipe using Filter:

	script.Echo("hello world").Filter(func(r io.Reader, w io.Writer) error {
		n, err := io.Copy(w, r)
		fmt.Fprintf(w, "\nfiltered %d bytes\n", n)
		return err
	}).Stdout()

Does this help?

[jesselang]

Does this help?

Yes, thanks. It brings me much closer to what I'm looking for. An undesired side effect of using Filter*() seems to be that a non-zero exit status from the program no longer propagates. Here's an example:

package main

import (
	"fmt"
	"io"

	"github.com/bitfield/script"
)

func main() {
	p := script.Exec("false")
	_, err := p.Stdout()
	fmt.Printf("%d %v\n", p.ExitStatus(), err)

	p = script.Exec("false").Filter(func(r io.Reader, w io.Writer) error {
		_, err := io.Copy(w, r)
		return err
	})
	_, err = p.Stdout()
	fmt.Printf("%d %v\n", p.ExitStatus(), err)
}

Output:

1 exit status 1
0 <nil>

Is this expected? I'd prefer to pass the exit status to the caller of this script. Thanks!

[bitfield]

No, that's a bug. Exec doesn't correctly detect the exit status of a failing command here. Would you mind opening a separate issue for this, and I'll fix it shortly.

[jesselang]

Thank you, @bitfield! I've created #148 to represent that bug as you requested.

Out of curiosity, is there an interest in providing a version of Exec() that doesn't interleave standard error? It seems like having the opportunity to keep a program's standard error separate (originally requested in #17 could provide a solution for #128. Doing so would remove the need for extra filtering code that could potentially be brittle and error-prone.

Something like:

script.Exec("ykman oath accounts code --single "someAccountId").WithStdout(buf).Stderr()

or

script.Exec("ykman oath accounts code --single "someAccountId").WithStderr(ignore).Stdout()

or

script.Exec("ykman oath accounts code --single "someAccountId").CombinedOutput() // similar to os/exec.Cmd.CombinedOutput

I'm just spitballing ideas here. Happy to dig in to more potential use cases to find a workable API.

[bitfield]

Yes, this is definitely something that a few people want, but we haven't so far come up with an elegant way to do that. The basic problem at the moment is that a pipe represents a single stream, whereas if you're dealing with stdout and stderr you now have two streams. Where do they go?

A good way to think about these things is always to start with a concrete example, I find. For example, you said:

I want to capture standard output (the code) as a string to pass to another pipe later in the script, while passing along standard error (preferably to os.Stderr, but os.Stdout would suffice) to provide the user with prompts and any error text

How would you prefer to write that as a script pipeline? Here's my first idea:

result, err := script.WithStderr(os.Stderr).Exec("foo").String()

This would mean that internally, the pipe can differentiate between the two streams, and Exec would send each stream to its appropriate destination. By default, they would both go to the pipe, as they do now. But by setting the pipe's stderr to something else beforehand, it would go there instead. (And it could be io.Discard, for example, if you wanted to just ignore it altogether.)

[jesselang]

Thanks for taking the time to discuss. My ideas are pretty naive as I haven't spent any time in the internals of how script implements its pipes. I certainly see the basic problem, and the additional complexity that comes with dealing with two streams vs. the one.

I think the idea you proposed would suit my use case and similar use cases very well, and it seems would also solve for #128. I think my only concern is one of consistency of ordering when comparing WithStdout() and a proposed WithStderr(). Would it confuse users that WithStderr() must be before Exec(), while WithStdout() can seemingly be placed anywhere in the pipe? Perhaps the nuance can be adequately covered with documentation, but I figured I'd bring it up for consideration.

I'd be happy to implement your idea in a PR (probably Jan 2023) and we could flesh out any remaining details there?

Thanks again for all your help!

[bitfield]

Let's see what other comments there are and whether anybody can improve on these ideas—more real-world programs using this feature would be very welcome.

[nchint]

Here's another example that has a similar problem, I want to have fzf run in the pipeline. For some background fzf outpts all its UI / interactive selection to stderr while the final selected line(s) are printed to stdout. I'm running into the same problem where I can't both display stderr and also process the output of stdout further down the pipeline

[bitfield]

What happens if you redirect stderr to /dev/null? For example:

script.Exec("sh -c 'fzf ... 2>/dev/null').Stdout()

[nchint]

So script.Exec("fzf").Stdout() works as expected, i.e. the fzf UI is shown on stderr, but any other sink i.e. script.Exec("fzf").Slice() results in both stderr and stdout going to the sink. For some background I'm writing a script that will fuzzy-find select python test files and then run them under pytest so I have something like

    script.FindFiles(".").
           MatchRegexp(pyFile).
           Reject(InitPyFile).
           Exec("fzf").
           // ... need to do more processing, but only with the stdout of fzf ...

[bitfield]

So, what happens if you try my suggested command instead of fzf in that pipeline?

[nchint]

Sorry, forgot to add it in yeah I tested your suggestion and I have the same issue where none of the stderr is shown, which I guess I expect with 2>/dev/null

[bitfield]

Sorry, I think I misunderstood what you were saying. I thought you wanted fzf's standard output to go to the pipe, for processing, but to discard the standard error output.

[nchint]

Ah I see now, here's a gif of running fzf https://cdn-media-1.freecodecamp.org/images/1*LTR424sh7y8E8rUzsUnFsQ.gif, the UI selection (i.e. the user typing in and selecting something) all of this UI is sent to stderr and I believe essential to using fzf at all. Finally the selected line(s) are sent to stdout. So I want a sink to only accept stdout, but stderr be still sent to os.Stderr so I can interact / select it

[bitfield]

Yes, I see—that's going to be a little difficult, I think. There is an issue (#128) to make it possible to send a command's stderr somewhere else (like the terminal). I'm not sure if Stdin actually sends keystrokes to interactive commands properly—you might like to try it and see. If so, combining those two things might make it work.

[nchint]

So I decided to experiment a bit with added support for arbitrary stderr here https://github.com/nchint/script/tree/EXEC-STDERR . Instead of adding a stderr to the Pipe struct here I just expanded the API to include more custom Exec commands. The way I saw it is Exec behaves similar to cmd 2>&1 | rest of PIpe... in the sense that stderr and stdout are by default combined, so to abstract that I made the following

ExecWithStdout(cmdLine string, stdout io.Writer)
ExecWithStderr(cmdLine string, stderr io.Writer)
ExecWithStdoutStderr(cmdLine string, stdout io.Writer, stderr io.Writer)

They feel a bit verbose and doesn't immediately add support for ExecForEach, but I'm curious what people's thoughts are before any PRs. I was able to get my fzf script working with the following

    testFiles, err := script.FindFiles(".").
        MatchRegexp(PyFileRegex).
        Reject(InitPyFile).
        ExecWithStderr("fzf -m", os.Stderr).
        Slice()