lordcodes/turtle

Running command that pipes to another command

lordcodes opened this issue · 11 comments

lordcodes commented

Describe the problem

Currently, there is no way via the API to pipe the output of a command into another command. For example to run something similar to:

echo '{"key":"value"}' | jq '.'

Describe your solution

  • Work out how to actually pipe one command into another using Process as currently it doesn't work. Likely the issue relates to the standard out and in being redirected/captured as streams.
  • Commands can be a structure rather than just executed immediately as a function like now.
  • A Pipeline structure can then be built from a series of Commands and then executed all together with each command piping into the next.

Checklist

Additional context

It looks like a change may be made with how the command(s) are prepared and built using ProcessBuilder so that they are executed correctly.

Workaround
Currently, you can run multiple commands by simply capturing the output of the first command into a Kotlin property and then running the second command using its value as input.

jmfayard commented

Maybe it's just me, but I prefer Kotlin typesafe syntax to Bash messy primitive obsession (everything is a string).

So I was thinking about a more functional approach where you create Commands data classes, combine them in a Pipeline data class, and only at the last step you execute it

/**
 * Pure function that represents but doesn't execute this command:
 *  $ find . -type f | grep '.kt' | xargs grep 'fun ' 2> /dev/null | wc -l
 */
fun measureKotlinFun(val output: File): Pipeline {
    val findFiles = Command("find", ".", "-type", "f")
    val onlyKotlinFiles = Command("grep", ".kt")
    val foreachSearchFun = Command("xargs", "grep", "fun ")
        .withRedirect(CommandRedirect(stderrIgnore = true))
    val countLines = Command("wc", "-l")
    val pipeline = findFiles `|` onlyKotlinFiles `|` foreachSearchFun `|` countLines
    return pipeline.withRedirect(CommandRedirect(stdout = output))
}

fun imperativeShell() {
    measureKotlinFun(File("fun.txt"))
        .executeCommand()
}

See https://github.com/jmfayard/turtle/blob/4291745b4552a65a4a7945403d7a84edf3e8c178/turtle/src/main/kotlin/com/lordcodes/turtle/Command.kt#L25-L36

What do you think?

lordcodes commented

Interesting idea. I actually investigated supporting piping in commands, but didn't get anywhere and actually realised it wouldn't be a good idea. You would be unable to use built-in commands and the result would be 'command' that is actually a chain of commands all as a String, difficult to parse and would be a bit confusing.

I really like the idea of making commands a structure that can then be executed, rather than just a function that is immediately executed. The existing commands would just create command and execute it internally. A pipeline structure can then contain a series of commands that can be executed as a chain.

The main blocker is working out how to implement the actual piping support with Process. I think it relates to checking for data being available on standard input and using that as input arguments if data is there.

jmfayard commented

The main blocker is working out how to implement the actual piping

I got it to work using https://github.com/zeroturnaround/zt-exec which seems like the right library to implement my Pipeline class

Proof of concept here: jmfayard@51aff7d?diff=split

lordcodes commented

Nice concept. We should be able to take the plumbing required to forward output of one to input of next without the dependency. Looks like a great starting point.

jmfayard commented

Yes but you would have to reinvent all the threading etc... yourself.
Personally I would start with the dependency as an implementation detail, if that works well you can always reinvent the wheel and remove it later.

lordcodes commented
jmfayard commented

Maybe I'm missing something, but to create a pipeline, don't you have to launch each program in a separate thread?
How would blocking calls work without that?

lordcodes commented

Maybe I just don't see it until I would try and work on it, however, there is no threading now and the pipeline would be running the commands sequentially. I don't see the difference to just running one command at a time in a process like they are now, but that each commands reads in from stream.

Anyway, realistically I'm not going to look into this or work on this anytime soon. If you would like this soon, then you are welcome to work on something. However, I'm really not a fan of introducing that library as a dependency, due to the fact it is a Java library with no recent changes. If threading and a lot of complexity is required to implement this, i.e. it isn't as simple as it seems to me 🤣, pipelines could be built as a separate library potentially.

jmfayard commented

Maybe I just don't see it until I would try and work on it, however, there is no threading now and the pipeline would be running the commands sequentially. I don't see the difference to just running one command at a time in a process like they are now, but that each commands reads in from stream.

There is a big difference between

yes "say yes indefinitely" | head -5

and

yes "Say yes indefinitely" > /tmp/yes.txt
head -5 < /tmp/yes.txt

The first pipeline ends in a few miliseconds and prints five times "Say yes indefinitely"

The second is an infinite loop that fills your SSD.

Here my version doesn't do an infinite loop

val inifiniteCommand = Command("yes", "say yes indefinitely")
val pipeline2 = inifiniteCommand `|` Command("head", "-5")
pipeline2.execcute()

jmfayard@51aff7d?diff=split

jmfayard commented
lordcodes commented

I agree. I have created #120 to track the Command abstraction.

  • ❤️1