Varallel

varallel is a simple and easy to use CLI tool to run commands in parallel. It is written in Vala, and with the use of GLib, it is cross-platform.

Build

Dependencies

Runtime:
- GLib
Build:
- Vala
- GLib
- Meson
- Ninja

Compile

Use meson to configure the build environment and build the project.

meson setup builddir
meson compile -C builddir

Usage

Help

Usage:
  varallel [OPTION…] command [:::|::::] [arguments]

Options:
  -h, --help                  Show this message
  -v, --version               Display version number
  -j, --jobs=n                Run n jobs in parallel
  -r, --colsep=EXPRESSION     Regex to split the argument
  -q, --quiet                 Hide subcommands output
  -s, --shell=SHELL           Manually set SHELL to run the command, set 'n' to disable to use any shell
  --hide-bar                  Hide progress bar
  --bar                       Show progress bar (Default behavior)
  --print-only                Only print the command but not run
  --color=WHEN                Enable color output, options are 'always', 'never', or 'auto'

Replacements in command:
  {}                          Input argument
  {.}                         Input argument without extension
  {/}                         Basename of input line
  {//}                        Dirname of input line
  {/.}                        Basename of input line without extension
  {#}                         Job index, starting from 1
  {3} {2.} {4/} {1/.} etc.    Positional replacement strings
  
For more information, or to report bugs, please visit:
    <https://github.com/wszqkzqk/varallel>

Explanation

{}
- Input argument. This replacement will be replaced by a full line read from the input source. The input source may be stdin (standard input), :::, or ::::.
{.}
- Input argument without extension. This replacement string will be replaced by the input with the extension removed. If the input argument contains . after the last / the last . till the end of the string will be removed and {.} will be replaced with the remaining.
  - E.g. foo.webp becomes foo, subdir/foo.webp becomes subdir/foo, sub.dir/foo.webp becomes sub.dir/foo, sub.dir/bar remains sub.dir/bar. If the input argument does not contain. it will remain unchanged.
{/}
- Basename of input argument. This replacement string will be replaced by the input with the directory part removed.
{//}
- Dirname of input argument. This replacement string will be replaced by the dir of the input argument.
{/.}
- Basename of Input argument without extension. This replacement string will be replaced by the input with the directory and extension part removed. It is a combination of {/} and {.}.
{#}
- Sequence number of the job to run. This replacement string will be replaced by the sequence number of the job being run. Starting from 1.
{3} {2.} {4/} {1/.} {5//} etc.
- Positional replacement strings. This replacement string will be replaced by the corresponding positional argument group. The first group is {1}, the second is {2}, and so on. Positional arguments can be combined with other replacement options.
:::
- Read the argument list from the command line.
::::
- Read the argument list from the files provided as the argument.
-j=n --jobs=n
- Run n jobs in parallel. The default value is the number of logical CPU cores.
-r=EXPRESSION --colsep=EXPRESSION
- User-defined regex to split the argument.
-q --quiet
- Hide subcommands output.
-s=SHELL --shell=SHELL
- Manually set SHELL to run the command, set it to n to disable to use any shell, and the subcommands will be spawned directly.
- If the --shell option is not provided, the program will use the shell specified in the SHELL environment variable in Unix-like systems, and directly spawn the subcommands in Windows.
- Note: If you use cmd.exe, powershell.exe and pwsh.exe in Windows, arguments contains unicode characters will not be handled correctly.`
--hide-bar
- Hide progress bar.
- If both --hide-bar and --bar are provided, the program will take the last option.
--bar
- Show progress bar. (Default behavior)
- If both --hide-bar and --bar are provided, the program will take the last option.
--print-only
- Only print the command but not run.
--color=WHEN
- Enable color output. Options are always, never, or auto.
- If --color is not provided, the program will use the auto option.
If there are more than one ::: or :::: in the command line, the replacement strings will be the Cartesian product of the argument lists.
- Example:
  - varallel echo ::: 1 2 ::: a b
  - The command will be run with the following arguments:
    - echo 1 a
    - echo 1 b
    - echo 2 a
    - echo 2 b

Examples

varallel can read the argument lists from pipes, files or command line.

Use pipes

varallel can read the argument lists from pipes.

seq 1 6 | varallel echo
varallel echo < <(seq 3 7)

Tip: You may also use varallel with find command:

find . -name '*.txt' | varallel cat
find . -type f | varallel 'echo "{}"'

Use command lines

varallel can read the argument lists from the command line.

varallel echo ::: 1 2 3 4 5 6
varallel 'echo "{.} {/} {#} {//}"' ::: /home/wszqkzqk ./README.md ~/Pictures/Arch_Linux_logo.svg
varallel 'echo "{1/} {0} {1.}"' ::: a b c d e f g ::: /home/wszqkzqk ./README.md ~/Pictures/Arch_Linux_logo.svg

Use files

Also, varallel can read the argument lists from files.

varallel echo :::: example.txt file*.txt