TaskRunNpmInstall.runNpmInstall() Process stdout probably not piped correctly on Windows10+Gradle, freezing pnpm

[mvysny]

Please see vaadin/base-starter-gradle#3 for more details.

The execution of the Gradle plugin halts under specific circumstances (empty pnpm cache) at:

"Execution worker for ':'" #178 prio=5 os_prio=0 tid=0x0000000018b4b000 nid=0x3d7c runnable [0x0000000029bbb000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.ProcessImpl.waitForInterruptibly(Native Method)
        at java.lang.ProcessImpl.waitFor(ProcessImpl.java:449)
        at com.vaadin.flow.server.frontend.TaskRunNpmInstall.runNpmInstall(TaskRunNpmInstall.java:374)
        at com.vaadin.flow.server.frontend.TaskRunNpmInstall.execute(TaskRunNpmInstall.java:118)
        at com.vaadin.flow.server.frontend.NodeTasks.execute(NodeTasks.java:634)
        at com.vaadin.gradle.VaadinBuildFrontendTask.runNodeUpdater(VaadinBuildFrontendTask.kt:132)
        at com.vaadin.gradle.VaadinBuildFrontendTask.vaadinBuildFrontend(VaadinBuildFrontendTask.kt:75)

My hypothesis is that pnpm is printing too much stuff into stdout which is not properly piped out. The buffer is rather small; when the buffer overflows, Windows will simply stop pnpm until the stdout buffer drains. Unfortunately there's nothing reading the stdout buffer and thus the buffer never drains.

I have no direct proof, but I have several items supporting this claim:

• When a Maven project is built on the same drive, it pre-populates the .pnpm-store folder. This will cause the gradle build to succeed, most probably because pnpm is not doing much download and will thus not overflow the stdout buffer.
• As opposed to Maven build, Gradle build prints nothing by default to stdout, not even when the --info switch is used.

Versions:

- Vaadin / Flow version: 17.0.0/4.0.0
- Java version: 11
- OS version: Windows 10
- Gradle plugin

[mvysny]

I confirm that on Windows 10 and Gradle, the call to inheritIO() will cause the process to block endlessly, never terminating. A very simple build.gradle.kts file reproduces the issue:

tasks.register("hello") {
    doLast {
        File("foo.txt").writeText((0..100000).joinToString())
        ProcessBuilder("cmd", "/c", "type", "foo.txt").inheritIO().start().waitFor()
    }
}

Run with gradlew hello and observe that it will never finish. The same block of code will complete successfully when launched as a main method:

fun main() {
    File("foo.txt").writeText((0..100000).joinToString())
    ProcessBuilder("cmd", "/c", "type", "foo.txt").inheritIO().start().waitFor()
}

Therefore the problem doesn't seem to lie in the code itself.

I think it's best to ask help in this regard from the Gradle guys.

[mvysny]

Asked on Gradle Forums: https://discuss.gradle.org/t/running-subprocess-on-windows-with-inheritio-never-completes/37523

Hopefully the Gradle folk will suggest a workaround which would make inheritIO() work on Gradle 👍 If not, hopefully they will suggest a workaround which doesn't include Gradle internal classes. In the worst case, we can use zt-exec and pump child stdout to System.stdout.

[mvysny]

The workaround is a bit silly - build any Vaadin 17 Maven project in production mode first, on the same Windows drive where the Gradle project resides. Maven will invoke pnpm which will then download+cache everything; Gradle project build will then no longer block since Gradle-invoked pnpm won't clutter the stdout buffer with download logs :-D

Generally, the idea of the workaround is to pre-populate D:\.pnpm-store first, before launching Gradle build. Please see vaadin/base-starter-gradle#3 for more details.

[caalador]

Yes in 2.3 we manually read the buffer:

            packageUpdater.log().debug("Output of `{}`:", commandString);
            StringBuilder toolOutput = new StringBuilder();
            try (BufferedReader reader = new BufferedReader(
                    new InputStreamReader(process.getInputStream(),
                            StandardCharsets.UTF_8))) {
                String stdoutLine;
                while ((stdoutLine = reader.readLine()) != null) {
                    packageUpdater.log().debug(stdoutLine);
                    toolOutput.append(stdoutLine);
                }
            }

where as in master branch we only inherit process = builder.inheritIO().start();

Seems like the fix #7325 has been missed in a merge as it has targeted master branch and 3.0 onward the used code is from #5506

The issue should be fixed by reapplying #7325 to master and 4.0 branches

[mvysny]

Can you cherry-pick the PR into the 4.0 branch so that I can test the fix with Gradle plugin?

[mvysny]

Related PR to the 4.0 branch: #8995

[mvysny]

Thank you, I confirm that newest flow from 4.0 fixes the issue in Gradle Plugin 👍

[swaechter]

@mvysny @caalador Thanks a lot for figuring this out + the fallback solution. I am writing a Gradle plugin too and it took me an hour to realize the problem

[mvysny]

@swaechter I've opened a Gradle bug report at gradle/gradle#16716 - it would be awesome if you could provide a bit more info and/or your use-case there as well 👍