r-lib/processx

Process hangs when using pipes with `stdout`, `stderr` for `latexmk -pdf` compilation

jemus42 opened this issue · 2 comments

jemus42 commented

I am attempting to orchestrate and keep track of dozens of latexmk compilation processes, and noticed that these processes never finish if I attempt to capture stdout or stderr with pipes.
It works when I log to temp files and read the contents back though, and I don't understand what's going on there.

Consider this minimal test.tex file:

\documentclass{article}

\begin{document}
Hello
\end{document}

Setting stdout and stderr to temp files

Works (mostly) as intended:

stdout_file <- tempfile()
stderr_file <- tempfile()

# Cleanup for next run
pc <- processx::process$new(command = "latexmk", args = c("-C"))
pc$wait()
if (file.exists("test.pdf")) file.remove("test.pdf")

p <- processx::process$new(
  command = "latexmk", args = c("-pdf", "test.tex"),
  stderr = stdout_file,
  stdout = stderr_file,
  echo_cmd = TRUE
)

p$wait()
p$get_exit_status()

Unfortunately all the output appears to be in stderr, but that's just a latexmk thing I assume

readLines(stdout_file)
character(0)
readLines(stderr_file)
 [1] "Rc files read:"                                                                                                                                                                                                                                                                                                                                      
 [2] "  NONE"                                                                                                                                                                                                                                                                                                                                              
[truncated for brevity]                                                                                                                                                                                                                                                                                                             
[30] "Latexmk: Log file says output to 'test.pdf'"                                                                                                                                                                                                                                                                                                         
[31] "Latexmk: All targets () are up-to-date"                                                                                                                                                                                                                                                                                                              
[32] ""

Setting stdout and stderr to pipes

Causes the process to hang indefinitely:

# Cleanup for next run
pc <- processx::process$new(command = "latexmk", args = c("-C"))
pc$wait()
if (file.exists("test.pdf")) file.remove("test.pdf")

p <- processx::process$new(
  command = "latexmk", args = c("-pdf", "test.tex"),
  stderr = "|",
  stdout = "|",
  echo_cmd = TRUE
)

# Hangs seemingly forever
p$wait()

Setting stdout and stderr to pipes with latexmk -C

It's specifically this compilation that's the issue, because piping with e.g. latexmk -C works fine:

p <- processx::process$new(
  command = "latexmk", args = c("-C"),
  stderr = "|",
  stdout = "|",
  echo_cmd = TRUE
)

p$wait()
p$read_all_output_lines()
[1] "Rc files read:"                                                    
[2] "  NONE"                                                            
[3] "Latexmk: This is Latexmk, John Collins, 4 Apr. 2023. Version 4.80."
[4] "No specific requests made, so using default for latexmk."          
[5] "Latexmk: Doing main (small) clean up for 'test.tex'"               
[6] ""                                                                  
p$read_all_error_lines()
character(0)

Session Info

Session Info 
system("latexmk --version")
#> Latexmk, John Collins, 4 Apr. 2023. Version 4.80

sessioninfo::session_info("processx")
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16)
#>  os       macOS Ventura 13.5.1
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Berlin
#>  date     2023-09-01
#>  pandoc   3.1.1 @ /System/Volumes/Data/Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package  * version date (UTC) lib source
#>  processx   3.8.2   2023-06-30 [1] CRAN (R 4.3.0)
#>  ps         1.7.5   2023-04-18 [1] CRAN (R 4.3.0)
#>  R6         2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#> 
#>  [1] /Users/Lukas/Library/R/arm64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────
gaborcsardi commented

Pipes have buffers, and if you don't read them out, then the process writing to them will indeed stop: https://github.com/r-lib/processx#standard-output-and-error

Always make sure that you read out the standard output and/or error of the pipes, otherwise the background process will stop running!

gaborcsardi commented

If you don't need to do anything until the process finishes, then processx::run() is the easiest to use.

If you want to parse the output while the process is running, then process$new() is best, here is an example: https://www.tidyverse.org/blog/2018/09/processx-3.2.0/#use-case-wait-for-an-external-process-to-be-ready

  • 👍1