shell

A mini Nim DSL to execute shell commands more conveniently.

Usage

With this macro you can simply write

shell:
  touch foo
  mv foo bar
  rm bar

which is then rewritten to something equivalent to:

execShell("touch foo")
execShell("mv foo bar")
execShell("rm bar")

where execShell is a proc around startProcess for normal compilation and gorgeEx when using NimScript.

Note: When using NimScript the given command is prepended by

&"cd {getCurrentDir()} && "

in order to switch the evaluation into the directory of the shell call. The same is achieved on the compiled backend by the poEvalCommand argument to startProcess.

See Full expansion of the macro below for more details and how to read the exit code of executed commands.

Most simple things should work as expected. See below for some known quirks.

`one` and `pipe`

By default each line in the shell macro will be handled by a different call to execShell. If you need several commands, which depend on the state of the previous, you may do so via the one command like so:

shell:
  one:
    mkdir foo
    cd foo
    touch bar
    cd ".."
    rm foo/bar

Similar to the one command, the pipe command exists. This concats the command via the shell pipe |:

shell:
  pipe:
    cat test.txt
    head -3

will produce:

execShell("cat test.txt | head -3")

Both of these can even be combined!

shell:
  one:
    mkdir foo
    pushd foo
    echo "Hallo\nWorld" > test.txt
    pipe:
      cat test.txt
      grep H
    popd
    rm foo/test.txt
    rmdir foo

will work just as expected, echoing Hallo in the shell.

Nim symbol quoting

NOTE: In a previous version this was done via accented quotes `. For the old behavior compile with -d:oldQuote.

Another important feature to make this library useful is quoting of Nim symbols.

This is handled via parenthesis () (if you need to run something in a subshell unfortunately that will have to be done with an explicit string now). Any tree in () is subject to quoting. That means if an identifier within () is preceded by a $, the symbol is unquoted. Note however that for the moment only a single variable may be quoted in each ().

The simplest case would be:

let name = "Vindaar"
shell:
  echo Hello from ($name)

which will perform the call:

execShell(&"echo Hello from {name}!")

and after the call to strformat.&:

execShell("echo Hello from Vindaar!")

Appending to a Nim identifier

Assuming we have a filename identifier and we want to convert some image from png to jpg with image magick. The simplest command should look like:

convert myimage.png myimage.jpg

This can be done in several ways.

Using dot expressions and no string literals:

let fname = "myimage"
shell:
  convert ($fname).png ($fname).jpg

Note that this is a special case. Continuing after a () quote without literal strings will only work for dot expressions. For instance:

let fname = "myimage"
shell:
  convert ($fname)".png" ($fname)".jpg"

will wrongly be converted to:

convert myimage .png myimage .jpg

which is obviously not what one would expect.

Using string literals:

let fname = "myimage"
shell:
  convert ($fname".pdf") ($fname".png")

In contrast to the wrong example shown above, this will work as expected.

This is especially useful for cases without dot expressions after the quoted nim identifier.

Appending a Nim identifier to a string literal

The other example would be appending a Nim identifier to a literal string. For instance in case we have a filename, which we create at run time and we wish to hand it to some command which takes an argument, which is must be given without a space like:

./myBin input --out=output

In this case one of the following ways works:

using `()` after a string literal:

let outfile = "myoutput.txt"
shell:
  ./myBin input "--out="($outfile)

If the () appears after the literal we can correctly generate the string without a space (in comparison to the case presented above when a string literal follows a ()).

For more predictable behavior, put the string literal also into

():

let outfile = "myoutput.txt"
shell:
  ./myBin input ("--out="$outfile)

General remark on predictability

NOTE: previously this section said to handle quoting + concatenation with strings both in the case of with and without space with () for the most predictable behavior. But that was a bad idea from my side! If you need spaces, simply put it outside the () and use a space!

The doAssert below is to be understood in the context of the shell macro. To summarize the above then:

let outfile = "myoutput.txt"
doAssert ("--out="$outfile) == &"--out={outfile}" # <- without space, ident after
doAssert "--out" ($outfile) == &"--out {outfile}" # <- with space, ident after
let fname = "myimage"
doAssert ($outfile".jpg") == &"{fname}.jpg" # <- without space, ident first
doAssert ($outfile) "image2" == &"{outfile} image2" # <- without space, ident first

NOTE 2: For the moment however, the () usage is restricted to a single string literal (or something that is convertible to a string via the stringify proc) and a single Nim identifier! This restriction will maybe be removed in the future.

This syntax also works for more complicated Nim expressions than a simple identifier:

const t = (a: "name", b: 5.5)
doAssert ("--out="$(t.a))
doAssert ("--out="$t.a)

both work. Of course t needn’t be a tuple. It can also be an object or even a function call, like for instance extracting a filename within a call:

import os, shell
let path = "/some/user/path/toAFile.txt"
shell:
  ./myBin ("--inputFile="$(path.extractFilename))

should produce:

./myBin --inputFile=toAFile.txt

Accented quotes

NOTE: In a previous version accented quotes were also used to quote Nim identifiers. That use case is now handled via parentheses. For the old behavior compile with -d:oldQuote.

Accented quotes allow you to hand raw strings.

Note: this has the downside of disallowing ` as a token to be handed to the shell. If you want to use the shell’s `, you need to put the appropriate command into quotation marks.

Raw strings

If you want to hand a literal string to the shell, you may do so by putting it into accented quotes:

echo `hello`

will be rewritten to

execShell("echo \"hello\"")

For a string consisting of multiple commands / words, put quotation marks around it:

echo `"Hello from Nim!"`

which will then also be rewritten to:

execShell("echo \"Hello from Nim!\"")

Assignment of results to Nim variables

Also useful is assignment of the result of a shell call to a Nim string. This can be done with the shellAssign macro. It is a little special compared to the shell and shellEcho macros. It only supports a single statement (*), which needs to be an assignment of a shell call of the syntax presented above to a Nim variable, such as:

var name = ""
shellAssign:
  name = echo Araq
assert name == "Araq"

Here the left name is the Nim variable (note: this is an exception of the Nim symbol quoting mentioned above!), whereas the right hand side is an arbitrary shell call, in this case a simple call to echo. The Nim variable will be assigned the result of the shell call, by being rewritten to:

var name = ""
name = asgnShell("echo Araq")
assert name == "Araq"

asgnShell is internally called by execShell mentioned above. asgnShell itself performs the calls to execCmdEx (or exec for NimScript).

(*): a single statement is not entirely precise, because the one and pipe operators can be used in combination with the assignment! For example the following is also possible:

var res = ""
shellAssign:
  res = pipe:
    seq 0 1 10
    tail -3
assert res == "8\n9\n10"

NimScript

This macro can also be used in NimScript! Instead of execCmdEx the nimscript.exec is used.

Known issues

Certain things unfortunately have to go into quotation marks. As seen in the one example above, the simple .. is not allowed.

Variable assignments in the shell need to be handed via a string literal:

shell:
  one:
    "a=`echo hello`"
    echo $a

Also if you need assignment via ‘:’ or ‘=’, put it also in quotation marks. Say you wish to compile a Nim program, you might want to do:

shell:
  nim c "--out:noTest" test.nim

In general, if in doubt you can just write strings or triple string (to pass a " to the shell).

Full expansion of the macro

As mentioned at the top of the README, the expansion shown is simplified (as a matter of fact it was as simple once, but has since become more complex).

The full expansion of the first example is:

discard block:
  var outputStr381052 = ""
  var exitCode381051: int
  if exitCode381051 ==
      0:
    let tmp381063 = execShell("touch foo")
    outputStr381052 = outputStr381052 &
        tmp381063[0]
    exitCode381051 = tmp381063[1]
  else:
    echo "Skipped command `" & "touch foo" &
        "` due to failure in previous command!"
  if exitCode381051 ==
      0:
    let tmp381064 = execShell("mv foo bar")
    outputStr381052 = outputStr381052 &
        tmp381064[0]
    exitCode381051 = tmp381064[1]
  else:
    echo "Skipped command `" & "mv foo bar" &
        "` due to failure in previous command!"
  if exitCode381051 ==
      0:
    let tmp381065 = execShell("rm bar")
    outputStr381052 = outputStr381052 &
        tmp381065[0]
    exitCode381051 = tmp381065[1]
  else:
    echo "Skipped command `" & "rm bar" &
        "` due to failure in previous command!"
  (outputStr381052, exitCode381051)

As can be seen from the expansion above, successive commands are only run, if the exit code of the previous command was 0, while the output is appended to the previous command’s output.

The normal shell command discards the return value of the block. If you want to keep it, use the shellVerbose macro:

let res = shellVerbose:
  someCommand

where res will be of type tuple[output: string, exitCode: string] according to the expansion above.

Debugging

In order to see what’s going on, you can either compile your program with the -d:debugShell flag, which will then echo the rewritten commands during compilation. Alternatively in order to avoid calling the commands immediately, you may use the shellEcho macro instead. It simply echoes the commands that would otherwise be run.

TC218/shell