Clojure library for shelling out / spawning sub-processes.
NOTE: When using process from babashka, this README assumes v1.0.168 or later.
In 90% of the use cases you will probably need shell and for the remaining use
cases you will probably need a combination of process and check. Start
reading the docs for those, skim over the rest and revisit the remaining functions when you need them.
See API docs as generated by quickdoc.
This library is included in babashka since 0.2.3 but is also intended as a JVM library.
The functions shell, process and exec take an optional map followed by one or more strings:
(require '[babashka.process :refer [shell process exec]])
(shell "ls" "-la") ;; no options
(shell "ls -la" "dir") ;; first string is tokenized automatically, more strings may be provided
(shell {:dir "target"} "ls" "-la")
(process {:in "hello"} "cat")
(exec {:extra-env {"FOO" "BAR"}} "bash")Previous versions of babashka process supported the (process ["prog" "arg"] {}) syntax. This syntax is no longer recommended, but is still supported to not
break existing programs.
Most commonly you will use shell. It executes a command and streams the output
to stdout and stderr while the process is running. The name shell comes from
"shelling out", but note that it does not invoke a bash/zsh/cmd.exe shell: it
just starts an external program.
user=> (shell "ls" "-la")
total 144
drwxr-xr-x@ 22 borkdude staff 704 Dec 4 13:39 .
drwxr-xr-x@ 75 borkdude staff 2400 Dec 3 14:18 ..
drwxr-xr-x@ 4 borkdude staff 128 Mar 10 2022 .circleci
drwxr-xr-x@ 5 borkdude staff 160 Mar 10 2022 .clj-kondo
drwxr-xr-x@ 50 borkdude staff 1600 Dec 3 20:55 .cpcacheThe first string argument to shell is tokenized automatically: "ls -la" is
broken up into "ls" and "-la", so (shell "ls -la") also works. This eases
the migration from existing bash scripts.
You can provide more arguments if you need to:
user=> (shell "ls -la" "src" "test")
src:
total 0
drwxr-xr-x@ 3 borkdude staff 96 Mar 10 2022 .
drwxr-xr-x@ 22 borkdude staff 704 Dec 4 14:13 ..
drwxr-xr-x@ 4 borkdude staff 128 Dec 4 14:01 babashka
test:
total 0
drwxr-xr-x@ 3 borkdude staff 96 Mar 10 2022 .
drwxr-xr-x@ 22 borkdude staff 704 Dec 4 14:13 ..
drwxr-xr-x@ 3 borkdude staff 96 Dec 4 14:01 babashkaThis is particularly handy when you want to supply commands coming from the command line:
(apply shell "ls -la" *command-line-args*)The shell function checks the command's exit code and throws if it is non-zero:
user=> (shell "ls nothing")
ls: nothing: No such file or directory
Execution error (ExceptionInfo) at babashka.process/check (process.cljc:113).To avoid throwing when the command's exit code is non-zero, use :continue true.
You will still see the error printed to stderr, but no exception will be thrown. This is convenient
when you want to handle the :exit code yourself:
user=> (-> (shell {:continue true} "ls nothing") :exit)
ls: nothing: No such file or directory
1Note that
:continue trueonly suppresses throwing an exception when the executed command's exit code is non-zero. Other exceptions can throw, for example, when the executable is not found.
To collect output as a :string, use the :out :string option as the first argument:
user=> (-> (shell {:out :string} "ls -la") :out str/split-lines first)
"total 144"To also capture stderr to a :string, add in the :err :string option:
user=> (-> (shell {:out :string :err :string} "git conpig user.name")
(select-keys [:out :err]))
{:out "borkdude\n", :err "WARNING: You called a Git command named 'conpig', which does not exist.\nContinuing in -1.1 seconds, assuming that you meant 'config'.\n"}To redirect stderr to stdout specify the :err :out option:
user=> (-> (shell {:out :string :err :out} "git conpig user.name") :out)
"WARNING: You called a Git command named 'conpig', which does not exist.\nContinuing in -1.1 seconds, assuming that you meant 'config'.\nborkdude\n"To change the working directory, use the :dir option:
user=> (-> (shell {:out :string :dir "src/babashka"} "ls -la") :out str/split-lines first)
"total 48"To add environment variables, use :extra-env:
user=> (-> (shell {:out :string :extra-env {"FOO" "BAR"}} "bb -e '(System/getenv \"FOO\")'") :out print)
"BAR"The shell function is a combination of process and deref and check. The
process function is the lower level function of this library that doesn't make
any opinionated choices:
- It does not provide a default for
:out,:inand:err: inshellthese default to:inheritwhich means: read and write from and to the console. Inprocessthey default to the default ofjava.lang.ProcessBuilder. - It does not wait until the process completes.
- It does not check the exit code and throw an exception.
Use process when you need to change one of the above and shell's options do
not support it. In practice this means: whenever you need async processing,
e.g. reading output from a process while it is running.
The return value of process implements clojure.lang.IDeref. When
dereferenced, it will wait for the process to finish and will add the :exit value.
user=> (-> (process "ls foo") deref :exit)
1The function check takes a process, waits for it to finish (so you can omit
deref) and returns it. When the exit code is non-zero, it will throw.
user=> (-> (process {:out :string} "ls") check :out str/split-lines first)
"API.md"
user=> (-> (process {:out :string} "ls foo") check :out str/split-lines first)
Execution error (ExceptionInfo) at babashka.process/check (process.clj:74).
ls: foo: No such file or directoryBoth :in, :out may contain objects that are compatible with clojure.java.io/copy:
user=> (with-out-str (check (process {:in "foo" :out *out*} "cat")))
"foo"
user=> (->> (with-out-str (check (process {:out *out*} "ls"))) str/split-lines (take 2))
("API.md" "CHANGELOG.md")The :out option also supports :string for collecting stdout into a string
and :bytes for getting the raw output as a byte array. You will need to
deref the process in order for the string or byte array to be there, since the
output can't be finalized if the process hasn't finished running:
user=> (-> @(process {:out :string} "ls") :out str/split-lines first)
"API.md"
user=> (-> @(process {:out :bytes} "head -c 10 /dev/urandom") :out seq)
(119 -43 -68 -64 -16 -56 32 45 86 56)Both shell and process support piping output from one process to the next
using but note that shell writes the output to the system's stdout by
default, so you have to provide it with {:out :string} for the next process to
capture the input, while process uses the default java.lang.ProcessBuilder
setting which defaults to writing to a stream:
user=> (let [stream (-> (process "ls") :out)]
@(process {:in stream
:out :inherit} "cat")
nil)
API.md
CHANGELOG.md
LICENSE
README.md
...Forwarding the output of a process as the input of another process can also be done with thread-first (->):
user=> (-> (process "ls")
(process {:out :string} "grep README") deref :out)
"README.md\n"To write to a file use :out :write and set :out-file to a file:
user=> (require '[clojure.java.io :as io])
nil
user=> (do @(process {:out :write :out-file (io/file "/tmp/out.txt")} "ls") nil)
nil
user=> (slurp "/tmp/out.txt")
"CHANGELOG.md\nLICENSE\nREADME.md..."or simply:
(do (shell {:out "/tmp/out.txt"} "ls") nil)To append to a file, use :out :append:
user=> (do @(process {:out :append :out-file (io/file "/tmp/out.txt")} "ls") nil)
nil
user=> (slurp "/tmp/out.txt")
"CHANGELOG.md\nLICENSE\nREADME.md..."Here is an example of a cat process to which we send input while the process
is running, then close stdin and read the output of cat afterwards:
(ns cat-demo
(:require [babashka.process :refer [process alive?]]
[clojure.java.io :as io]))
(def catp (process "cat"))
(alive? catp) ;; true
(def stdin (io/writer (:in catp)))
(binding [*out* stdin]
(println "hello"))
(.close stdin)
(slurp (:out catp)) ;; "hello\n"
(:exit @catp) ;; 0
(alive? catp) ;; falseHere is an example where we read the output of bb -o -e '(range)', an infinite
stream of numbers, line by line and print it ourselves:
(require '[babashka.process :as p :refer [process destroy-tree]]
'[clojure.java.io :as io])
(def number-stream
(process
{:err :inherit
:shutdown destroy-tree}
"bb -o -e '(range)'"))
(with-open [rdr (io/reader (:out number-stream))]
(binding [*in* rdr]
(loop [max 10]
(when-let [line (read-line)]
(println :line line)
(when (pos? max)
(recur (dec max)))))))
;; kill the streaming bb process:
(p/destroy-tree number-stream)The :pre-start-fn option can be used to report commands being run:
(require '[babashka.process :refer [process]])
(doseq [file ["LICENSE" "CHANGELOG.md"]]
(-> (process
{:out :string
:pre-start-fn #(apply println "Running" (:cmd %))}
"head" "-1" file)
deref :out println))
Running head -1 LICENSE
Eclipse Public License - v 1.0
Running head -1 CHANGELOG.md
# Changelogsh is a convenience function around process which sets :out and :err to
:string and blocks automatically, similar to clojure.java.shell/sh:
user=> (def config {:output {:format :edn}})
#'user/config
user=> (-> (sh ["clj-kondo" "--lint" "src"]) :out slurp edn/read-string)
{:findings [], :summary {:error 0, :warning 0, :info 0, :type :summary, :duration 34}}All of shell, process and sh support tokenization on the first string argument using tokenize:
user=> (require '[babashka.process :refer [sh tokenize]])
nil
user=> (tokenize "hello there")
["hello" "there"]
user=> (-> (sh "echo hello there") :out)
"hello there\n"user=> (-> (sh {:in "(inc)"} "clj-kondo --lint -") :out)
"<stdin>:1:1: error: clojure.core/inc is called with 0 args but expects 1\nlinting took 10ms, errors: 1, warnings: 0\n"Note that check will wait for the process to end in order to check the exit
code. When the process has lots of data to write to stdout, it is recommended to
add an explicit :out option to prevent deadlock due to buffering. This example
will deadlock because the process is buffering the output stream but it's not
being consumed, so the process won't be able to finish:
user=> (-> (process {:in (apply str (repeat 1000000 "hello\n"))} "cat") check :out count)The way to deal with this is providing an explicit :out option so the process
can finish writing its output:
user=> (-> (process {:out :string :in (apply str (repeat 1000000 "hello\n"))} "cat") check :out count)
6000000The :env option replaces your entire environment with the provided map. To add environment variables you can use :extra-env instead:
:extra-env {"FOO" "BAR"}Windows TIP: Unlike in the CMD and Powershell shells, environment variable names are case sensitive for
:extra-env. For example,"PATH"will not update the value of"Path"on Windows. Here's an example of a babashka task that understands this nuance.
:env TIP: An OS might have default environment variables it always includes. For example, as of this writing, Windows always includes
SystemRootand macOS always includes__CF_USER_TEXT_ENCODING.
The pipeline function returns a
sequential
of processes from a process that was created with -> or by passing multiple
objects created with pb:
user=> (require '[babashka.process :refer [pipeline pb process check]])
nil
user=> (mapv :cmd (pipeline (-> (process "ls") (process "cat"))))
[["ls"] ["cat"]]
user=> (mapv :cmd (pipeline (pb "ls") (pb "cat")))
[["ls"] ["cat"]]To obtain the right-most process from the pipeline, use last (or peek):
user=> (-> (pipeline (pb "ls") (pb "cat")) last :out slurp)
"LICENSE\nREADME.md\ndeps.edn\nsrc\ntest\n..."Calling pipeline on the right-most process returns the pipeline:
user=> (def p (pipeline (pb "ls") (pb "cat")))
#'user/p
user=> (= p (pipeline (last p)))
trueTo check an entire pipeline for non-zero exit codes, you can use:
user=> (run! check (pipeline (pb "ls foo") (pb "cat")))
Execution error (ExceptionInfo) at babashka.process/check (process.clj:37).
ls: foo: No such file or directoryAlthough you can create pipelines with ->, for some applications it may be
preferable to create a pipeline with pipeline which defers to
ProcessBuilder/startPipeline. In the following case it takes a long time
before you would see any output due to buffering.
(future
(loop []
(spit "log.txt" (str (rand-int 10) "\n") :append true)
(Thread/sleep 10)
(recur)))
(-> (process "tail" "-f" "log.txt")
(process "cat")
(process {:out :inherit} "grep "5"))The solution then it to use pipeline + pb:
(pipeline (pb "tail" "-f" "log.txt")
(pb "cat")
(pb {:out :inherit} "grep" "5"))The varargs arity of pipeline is only available in JDK9 or higher due to the
availability of ProcessBuilder/startPipeline. If you are on JDK8 or lower, the
following solution that reads the output of tail line by line may work for
you:
user=> (require '[clojure.java.io :as io])
nil
(def tail (process {:err :inherit} "tail" "-f" "log.txt"))
(def cat-and-grep
(-> (process {:err :inherit} "cat")
(process {:out :inherit
:err :inherit} "grep 5")))
(binding [*in* (io/reader (:out tail))
*out* (io/writer (:in cat-and-grep))]
(loop []
(when-let [x (read-line)]
(println x)
(recur))))Another solution is to let bash handle the pipes by shelling out with bash -c.
On macOS & Linux, programs are resolved the way you expect:
aresolves against the systemPATH./aresolves against:dirif specified, otherwise the current working directory/some/absolute/aresolves absolutely
In all cases, the working directory for a is :dir, if specified, otherwise your current working directory.
Windows executable files have extensions, which, if not specified, are resolved in order: .com,.exe,.bat,.cmd.
Programs are resolved in directories using the same rules as macOS, Linux, and Windows PowerShell.
Windows .ps1 TIP: Babashka process will never resolve to, and cannot launch,
.ps1scripts directly. To launch a.ps1script, you must do so through PowerShell. Example:(p/shell "powershell.exe -File .\\a.ps1")
Windows TIP: If you prefer a more CMD Shell-like experience where programs are resolved first in the current working directory, then on the
PATH, and are OK with the security implications of doing so, you can override the default:program-resolverwith your own.
If clojure.java.shell works for your purposes, keep using it. But there are
contexts in which you need more flexibility. The major differences compared with
this library:
shis blocking,processmakes blocking explicit viaderefshfocuses on convenience but limits what you can do with the underlying process,processexposes as much as possible while still offering an ergonomic APIprocesssupports piping processes via->orpipelineshoffers integration withclojure.java.io/copyfor:in,processextends this to:outand:err
Clojure 1.12.0 features a new clojure.java.process
namespace. babashka.process predates it, but the API is very similar, although defaults will differ.
Note that the exec function in babashka.process does something very
different than the same-named function in clojure.java.process: in
babashka.process it replaces the parent process via a Unix exec call, while
in clojure.java.process/exec the process is launched as a child process.
Other notable differences:
clojure.java.processdoes not do any tokenization, so it doesn't support passing a string like"ls -la"clojure.java.processdoes not have any Windows-specific support.
Because process spawns threads for non-blocking I/O, you might have to run
(shutdown-agents) at the end of your Clojure JVM scripts to force
termination. Babashka does this automatically.
When pretty-printing a process, by default you will get an exception:
user=> (require '[babashka.process :refer [process]])
nil
user=> (require '[clojure.pprint :as pprint])
nil
user=> (pprint/pprint (process "ls"))
Execution error (IllegalArgumentException) at user/eval257 (REPL:1).
Multiple methods in multimethod 'simple-dispatch' match dispatch value: class babashka.process.Process -> interface clojure.lang.IDeref and interface clojure.lang.IPersistentMap, and neither is preferredThe reason is that a process is both a record and a clojure.lang.IDeref and
pprint does not have a preference for how to print this. The recommended solution is to require the babashka.process.pprint namespace, which will define a pprint implementation for a Process record:
user=> (require '[babashka.process.pprint])
nil
user=> (pprint/pprint (process "ls"))
{:proc
#object[java.lang.ProcessImpl 0x1d61a348 "Process[pid=43771, exitValue=\"not exited\"]"],
:exit nil,
...You can combine this library with promesa
in the following way. This requires :exit-fn which was released in version
0.2.10.
(require '[babashka.process :as proc]
'[promesa.core :as prom])
(defn process
"Returns promise that will be resolved upon process termination. The promise is rejected when the exit code is non-zero."
[opts & cmd]
(prom/create
(fn [resolve reject]
(let [exit-fn (fn [response]
(let [{:keys [exit] :as r} response]
(if (zero? exit)
(resolve r)
(reject r))))]
(apply proc/process (assoc opts :exit-fn exit-fn) cmd)))))
(prom/let [ls (process
{:out :string
:err :inherit}
"ls")
ls-out (:out ls)]
(prn ls-out))Copyright © 2020-2022 Michiel Borkent
Distributed under the EPL License. See LICENSE.