How to get output before input close

[Saafo]

Maybe it's a problem related to NSProcess, but I just want to discuss with you(I have much less research on NSProcess)

The scenario is I have a steaming xcodebuild output, and I want to pass it to xcbeautify/xcpretty and get the result streamingly. The demo code is like:

        let echoProcess = try Command.findInPath(withName: "cat")!
            .setStdin(.read(fromFile: "/users/Saafo/Downloads/test.log"))
            .setStdout(.pipe)
            .spawn()
        let addProcess = try Command.findInPath(withName: "xcbeautify")!
            .setStdin(.pipe)
            .setStdout(.pipe)
            .spawn()
        for try await result in echoProcess.stdout.lines {
            print("file: \(result)")
            addProcess.stdin.write(result)
        }
        let result = Task {
            print("stdout start")
            for try await result in addProcess.stdout.lines {
                print("result: \(result)")
            }
            print("stdout end")
            return 0
        }
        try await Task.sleep(nanoseconds: 1 * 1000 * 1000 * 1000)
        print("stdin end")
        addProcess.stdin.close() // TODO: how to get output before close
        print(try await result.value)
        print("program end")

And the content of test.log is:

CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')

The output of my code is:

file: CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')
stdout start
stdin end
result: [�[36mTemp�[0m] �[1mCompiling�[0m ContentView.swift
stdout end
0
program end
Program ended with exit code: 0

The issue is, I gave the input to xcbeautify and sleep for 1 second. The output from xcbeautify is expected to print directly, however, it seems to wait for stdin to close and then gave all outputs. I want get the output from xcbeautify before input close since the real xcodebuild may take a long time, I need to make it streamingly.

Really expected to your kindly help! I've stuck with this issue for a long time.

[Zollerboy1]

AFAICS, this is a problem with the interaction between xcbeautify and Foundation.Process. I wrote the following test and it works as expected for cat:

let process = try Command.findInPath(withName: "cat")!
    .setStdin(.pipe)
    .setStdout(.pipe)
    .spawn()

let result = Task.detached {
    print("stdout start")
    for try await result in process.stdout.lines {
        print("result: \(result)")
    }
    print("stdout end")
    return 0
}

print("stdin start")
process.stdin.write("CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')\n")
print("stdin write")

try await Task.sleep(nanoseconds: 1 * 1000 * 1000 * 1000)

process.stdin.write("CompileSwift normal x86_64 /Users/ci/132/Temp/Source/DetailView.swift (in target 'Temp' from project 'Temp')\n")
print("stdin write")

try await Task.sleep(nanoseconds: 1 * 1000 * 1000 * 1000)

process.stdin.close()
print("stdin end")

print(try await result.value)
print("program end")

This outputs the following:

stdin start
stdout start
stdin write
result: CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')
stdin write
result: CompileSwift normal x86_64 /Users/ci/132/Temp/Source/DetailView.swift (in target 'Temp' from project 'Temp')
stdin end
stdout end
0
program end

However, if I change cat to xcbeautify, I get the following output:

stdin start
stdout start
stdin write
stdin write
stdin end
result: [Temp] Compiling ContentView.swift
result: [Temp] Compiling DetailView.swift
stdout end
0
program end

I don't know exactly, how to solve this problem. It's also not an inherent problem of xcbeautify, as it works normally, when not creating a pipe for stdout, but instead just inheriting the stdout of the parent process, i.e. this works just fine:

let process = try Command.findInPath(withName: "xcbeautify")!
    .setStdin(.pipe)
    .spawn()

print("stdin start")
process.stdin.write("CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')\n")
print("stdin write")

try await Task.sleep(nanoseconds: 1 * 1000 * 1000 * 1000)

process.stdin.write("CompileSwift normal x86_64 /Users/ci/132/Temp/Source/DetailView.swift (in target 'Temp' from project 'Temp')\n")
print("stdin write")

try await Task.sleep(nanoseconds: 1 * 1000 * 1000 * 1000)

process.stdin.close()
print("stdin end")
print("program end")

[Saafo]

Today I've done these attempts, and found maybe it's not (directly) related to xcbeautify:

1. replace cat/xcbeautify with my custom swift file xcb.swift, which can be considered to have the same I/O behavior as xcbeautify, and the content is:

while let line = readLine() {
    print("xcb:" + line)
}

and I set the process like:

let addProcess = try Command.findInPath(withName: "swift")!
    .addArgument("/path/to/xcb.swift")
    .setStdin(.pipe)
    .setStdout(.pipe)
    .spawn()

And it continue to have the issue. I've also change to the shell version, like:

#!/bin/sh

while read line; do
	input_str=$(echo $line | tr -d '\n')
	output_str="aaa $input_str"
	echo "$output_str"
done

Which also have the issue.

2. I tried to reproduce your cat process, when I copied the whole codes, the results are the same. However, if I use two cat processes chaining together, the issue also happens:

        let echoProcess = try Command.findInPath(withName: "cat")!
            .setStdin(.read(fromFile: "/users/Saafo/Downloads/test.log"))
            .setStdout(.pipe)
            .spawn()
        let addProcess = try Command.findInPath(withName: "cat")!
            .setStdin(.pipe)
            .setStdout(.pipe)
            .spawn()
        for try await result in echoProcess.stdout.lines {
            print("file: \(result)")
            addProcess.stdin.write(result)
        }
        let result = Task {
            print("stdout start")
            for try await result in addProcess.stdout.lines {
                print("result: \(result)")
            }
            print("stdout end")
            return 0
        }
        try await Task.sleep(nanoseconds: 1 * 1000 * 1000 * 1000)
        print("stdin end")
        addProcess.stdin.close()
        print(try await result.value)
        print("program end")

output:

file: CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')
stdout start
stdin end
result: CompileSwift normal x86_64 /Users/ci/132/Temp/Source/ContentView.swift (in target 'Temp' from project 'Temp')
stdout end
0
program end
Program ended with exit code: 0

[Zollerboy1]

Hmm. Well there is not much that I can do, I believe. If you try to use raw Foundation.Process to do the same thing, the same problem arises. Maybe I'm using the API wrong and if it's used correctly, it works, but otherwise I cannot really solve this issue.

[Saafo]

@Zollerboy1 You're right, I began with Foundation.Process and met the issue. Then I thought maybe I'm using the API incorrectly and began using this tool, but ending up with the same issue. I'll try to write a demo with Foundation.Process and post this issue to stackoverflow and see if I can get some help from there. Thanks again with your kindly help!