% MParallel
MParallel is a batch processor with multi-threading support, i.e. it will run multiple tasks in parallel. This can be very useful, not only, to take full advantage of multi-processor (multi-core) machines.
Created by LoRd_MuldeR <mulder2@gmx>. Please check http://muldersoft.com/ for news and updates!
MParallel can be invoked in three different forms:
MParallel.exe [options] <command_1> : <command_2> : ... : <command_n>
MParallel.exe [options] --input=commands.txt
GenerateCommands.exe [parameters] | MParallel.exe [options] --stdin
The first form takes commands directly from the command-line, delimited by colon (:) characters. The second from reads commands from a text file, line by line. And the third form reads commands from the standard input stream (stdin), which is typically used to process the output from another program using a pipe (|). The three forms can be combined freely.
The best way to get started with MParallel is looking at some examples:
This basic example uses MParallel to run multiple "ping" commands in parallel:
MParallel.exe --count=3 ping.exe -n 16 fsf.org : ping.exe -n 16 gnu.org : ping.exe -n 16 w3c.org
Note how the distinct commands and their related arguments are delimited by colon (:) characters. Also note how the command-line options specific to MParallel have to go before the very first command string.
A slightly more advanced example, using a command pattern to express the above command-line more elegantly:
MParallel.exe --count=3 --pattern="ping.exe -n 16 {{0}}" fsf.org : gnu.org : w3c.org
Note that, in this example, the {{0}} placeholder will be replaced with the corresponding command tokens.
It is also possible to read your commands (or the parameters for your command pattern) from a text file:
MParallel.exe --count=3 --input=my_commands.txt
The content of the file "my_commands.txt" may look like this, for example:
ping.exe -n 16 fsf.org
ping.exe -n 16 gnu.org
ping.exe -n 16 w3c.org
Now let's read the output of the "dir" command to copy all "*.jpg" files in the current directory to "*.png":
dir /b *.jpg | MParallel.exe --shell --stdin ---pattern="copy {{0}} {{0:N}}.png"
Note that here we need to use the --shell option, because copy is a built-in shell function. Also note that we would need to add --no-split-lines and --auto-wrap in order to correctly handle file names containing spaces!
Please refer to the included Example_1.cmd script for an in-depth MParallel example on how to convert all JPEG files in a directory to the PNG format, in parallel, using ImageMagick. You will need ImageMagick for Windows.
Also, refer to the included Example_2.cmd script for an in-depth MParallel example on how to convert all AVI files in a directory to the MP4 format, in parallel, using FFmpeg/Libav. You will need FFmpeg builds for Windows.
The following MParallel options are currently available:
Run at most N instances in parallel. MParallel will start N commands in parallel, provided that there are at least N commands in the queue. If there are less than N commands in the queue, it will start as many commands in parallel as there are in the queue. If there are more than N commands in the queue, MParallel will start the first N commands in parallel and, at each time that any of the running commands completes, it will start the next command. This way, always N commands will be running in parallel, unless the queue is running empty. Note that N defaults to the number of available processors (CPU cores), if not specified explicitly – taking into account the processor affinity mask.
Generate commands from the specified PATTERN string. If a PATTERN string has been specified, the commands passed to MParallel on the command-line, read from a file or read from the STDIN stream will not be executed "as-is". Instead, any given command-tokens will then be interpreted as input parameters for transforming the given PATTERN string.
The PATTERN string should contain placeholders in the {{k}} form. Placeholders of that from will be replaced by the k-th command-token. Note that the placeholder indices k are zero-based, i.e.use {{0}}, {{1}}, {{2}} and so on. In addition, if the k-th command-token represents a valid file path, the placeholders {{k:F}}, {{k:D}}, {{k:P}}, {{k:N}} and {{k:X}} will be replaced with the file's full path (expanded relative to working directory), drive letter (with trailing colon), directory name (with trailing backslash), file name (without extension) and extension (including dot), respectively.
Note that if the PATTERN string contains any whitespace characters, the PATTERN string as a whole needs to be wrapped in quotation marks (e.g.--pattern="foo bar"). Also note that any quotation marks inside the PATTERN string need to be escaped by a \" sequence (e.g. --pattern="foo \"{{0}}\""). However, using the --auto-wrap option can simplify building PATTERN strings. Finally note that any excess command-tokens will be discarded by MParallel!
Set the command separator to SEP. The separator string is used to delimit the distinct commands, when they are passed to MParallel on the command-line. By default, a single colon character (:) is used as separator, but any suitable character sequence may be specified here. Note that SEP is not used for reading commands from a file or from the STDIN. When reading from a file or from the STDIN, there must be one command per line.
Read additional commands from the specified FILE. The specified FILE needs to be a plain text file, containing one command per line. Each line is interpreted like a full command-line. Using separators within a line is not required or supported, because commands are delimited by line breaks. If the file contains any characters other than plain US-ASCII, it is expected to be in UTF-8 text encoding by default. Use the option --utf16 in order to force UTF-16 encoding.
Read additional commands from the STDIN stream. The data on STDIN needs to be plain text, containing one command per line. Each line is interpreted like a full command-line. Using separators within a line is not required or supported, because commands are delimited by line breaks. If the stream contains any characters other than plain US-ASCII, it is expected to be in UTF-8 text encoding by default. Use the option --utf16 in order to force UTF-16 encoding.
This option is typically used to process lines produced by other programs or by shell functions. In the shell (cmd.exe) the STDOUT of another program can be "connected" to the STDIN of MParallel using a pipe operator (|). Note, however, that shell functions like dir may not output UTF-8 by default. Set the shell to UTF-8 mode (chcp 65001) in advance!
Save logfile to FILE. The logfile contains information about all processes that have been created an the result. By default, no logfile will be created. If the logfile already exists, MParallel appends to the existing file. Log output format is:
[YYYY:MM:DD hh:mm:ss] <log_message>
Redirect the STDOUT and STDERR streams of each sub-process to a file. MParallel will create a separate output file for each process in the PATH directory. File names are generated according to the YYYYMMDD-HHMMSS-NNNNN.log pattern. Note that directory PATH must be existing and writable. Also note that all redirected outputs do not appear in the console!
Automatically wrap all tokens that contain any whitespace characters in quotation marks. This applies to the expansion of placeholders, when the --pattern option is used. For example, if the N-th command token contains foo bar, then {{N}} will be replaced by "foo bar" instead of foo bar. This option has no effect, if --pattern is not used.
Ignore whitespace characters when reading commands from a file. By default, when MParallel reads commands from a file or from the STDIN stream, each input line will be processed like a full command-line. This means that tokens within each line are whitespace-delimited, unless wrapped in quotation marks. If this option is set, no command-line splitting is performed on the input lines. Instead, each input line will be treated like one unbroken string.
Start each command inside a new sub-shell (cmd.exe). Running each command in a new sub-shell implies a certain overhead, which is why this behavior is disabled by default. However, you must use this option, if your command uses any built-in shell functions, such as echo, dir or copy. You also must use this option, if your command contains any shell operators, such as the pipe operator (|) or one of the redirection operators (>, <, etc).
Kill processes after TIMEOUT milliseconds. By default, each command is allowed to run for an infinite amount of time. If this option is set, a command will be aborted if it takes longer than the specified timeout interval. Note that (by default) if a command was aborted due to timeout, other pending commands will still get a chance to run.
Run the commands (sub-processes) with the specified process priority. This can be one of the following values:
- 0: Lowest priority
- 1: Lower than normal priority
- 2: Default priority
- 3: Higher than normal priority
- 4: Highest priority
Run each sub-process in a separate console window. By default, all sub-processes are connected to the same console window as the main MParallel process. Thus, output from all processes will appear in the same console window, in an "interleaved" fashion. With this option set, each sub-process gets a separate console window.
Note that this option is mutually exclusive to the --out-path=<PATH> option. That is because, when output redirection is in effect, the detached console windows would just end up being completely blank.
Abort batch, if any command failed to execute. By default, if any command failed, e.g. because the process could not be created or because it returned a non-zero exit code, other pending commands will still get a chance to run. If this option is set, the whole batch (queue) will be aborted, as soon as one command has failed.
Do not add new sub-processes to job object. By default, MParallel adds all new sub-processes to a Job Object, which makes sure that all sub-processes will die immediately when the MParallel process is terminated. If this option is set, sub-processes are not added to the Job Object and may continue running after the MParallel was terminated.
Do not apply priority boost to MParallel process. By default, the MParallel process will run with somewhat higher priority than the sub-processes. Also the system timer precision will be improved. This does not aim to give the MParallel process more CPU time (it needs very few CPU time anyway!), but to improve the delays in process creation and termination.
Discard the STDOUT and STDERR streams of any sub-processes. This means that any sub-process outputs will neither be visible in the console, nor will they be saved to a file. This option is mutually exclusive to the --out-path option.
Do not check the exit code of sub-processes. By default, MParallel checks the exit code of each sub-process. It assumes that the command has failed, if the process returned a non-zero exit code. If any command failed, this will be reported and, if --abort is set, any pending commands will not be executed. Setting this option causes MParallel to ignore exit codes. However, a command is still considered to have failed, if the processes could not be created.
Interpret the lines read from a file or from the STDIN stream with UTF-16 text encoding. By default, MParallel will interpret the lines read from a file or from the STDIN with UTF-8 text encoding. In most cases UTF-8 is what you want.
Play a notification sound, as soon as all commands have been completed (or failed) and MParallel is about exit.
Disable all textual messages, also known as "silent mode". Note that fatal error messages may still appear under some circumstances. Also note that this option is mutually exclusive with the --trace option.
Disables colored textual output to the console. By default, when MParallel is writing textual messages to the console window, it will apply the appropriate colors to them. Colors are always disabled when redirecting STDERR to a file.
Enable more diagnostic outputs. This will print, e.g., the full command-line and the exit code for each task, which can be helpful for debugging purposes. Note that this option is mutually exclusive with the --silent option.
Print the help screen, also known as "manpage".
MParallel returns max(exitcode_1, exitcode_2, ..., exitcode_N) as its exit code, where exitcode_i is the exit code that was returned by i-th sub-process. In general, a zero exit code indicates that all commands completed successfully, while a non-zero exit code indicates that at least one command has failed. Fatal errors are indicated by a 666 exit code.
Optionally, MParallel can read default options from a configuration file. Any options read from the configuration file will override the "built-in" defaults. However, options passed on the command-line will always have precedence!
In order to read options from a configuration file, create a text file in the same directory where the MParallel executable is located and with the same base name as the MParallel executable, but with a .ini file extension. If, for example, your executable path is C:\Gizmo\MParallel.exe, create the configuration file at C:\Gizmo\MParallel.ini.
The configuration file may contain any MParallel option that is available on the command-line. It should start with a single [MParallel] header line (any lines before the header are ignored!) and, from that point on, contain one option per line. Options are specified in the <name>=<value> format. However, the -- prefix is neither required nor allowed.
Note that if the configuration file contains any characters other than plain US-ASCII, it is expected to be in UTF-8 text encoding. Also note that boolean options can be overwritten using the --some-option={0,1} syntax.
A simple configuration file may look like this:
[MParallel]
shell
priority=4
…which is equivalent to passing --shell and --priority=4 on the command-line. Using this example configuration file, it is possible to disable the --shell option (now set by default) by passing --shell=0 on the command-line.
MParallel source code is available from the official Git mirrors at:
MParallel is released under the GNU General Public License, Version 2.
MParallel - Parallel Batch Processor
Copyright (C) 2016 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
http://www.gnu.org/licenses/gpl-2.0.html
Using Application Lightning Icon, created by Fatcow Web Hosting
License: CC Attribution 4.0 license
(Backlink to http://www.fatcow.com/free-icons required)
-
Added support for reading default options from a configuration file (
MParallel.ini) -
Added new option
--notifyto play a sound as soon as all commands are completed -
Apply priority boost to MParallel "main" process by default (disabled by
--no-boost) -
Documentation improvements
-
Implemented progress display in console title and console icon (disabled by
--silent) -
Open the log file in "shared" mode so that it is readable while MParallel is still running
-
More code clean up and refactoring
- Fixed various regressions.
-
Implemented colored console output, enabled by default, disable with
--no-colorsoption -
Added new option
--utf16to treat file (or STDIN) inputs as UTF-16 -
Added two example scripts, see
Example_1.cmdandExample_2.cmdfor details -
Build binaries with VS2010 to workaround the line reading bug currently present in VS2015
-
Massive code refactoring and clean-up
- This is the initial release of the MParallel application.
e.o.f.