in long command strings, 8,262th character consistently missing

[jgmeyerucsd]

I'm passing hundreds of commands to mParallel, and I just noticed that one of the processes always fails, and this results from omission of exactly the 8,262th character from each string when the command is passed to the console. Please see the attached example of the input batch where the commands are specified correctly, and the attached log where the command actually run is missing a character that causes the file path to be jibberish.

peptideProphet.example.txt
mparallel.log.txt

[lordmulder]

Hello,

it's a bit hard to understand from your logs what is the expected outcome and what actually happens. So, could you please provide a minimal example – full command-line and, if required, input files – that I can use to reproduce the issue? Also, what version exactly are you running?

Note: There was an issue with VS2015 – or, more specifically, the UCRT (Universal CRT) that VS2015 uses – that sometimes messed up the input, which is why I reverted to VS2010 in latest build.

Regards,
MuldeR

[jgmeyerucsd]

Thanks for being responsive, but I think this is out of your control as a windows command length limit. I'm using version 1.0.4 - apparently there is a hard limit about how many characters a single command can be in windows cmd.exe. The limit is something like 8,190 characters. To avoid this I changed my java code that creates my .bat to check if the stringbuffer.length()> 7000 and if so, write one mparallel line and start a new one.

Still, if you want to reproduce it, you can make a mparallel command that submits 10,000 ping requests in a batch file thereby containing more than 8,262 characters and you'll notice that one of them has non-zero exit because a character is missing.

[lordmulder]

You mean that your MParallel command-line exceeds ~8000 characters? 😨

In this case you really should do things differently in order not run into the command-line length limit!

Your are probably calling MParallel like this at the moment (right?):

MParallel.exe [options] <command_1> : <command_2> : ... : <command_n>

A way better method for a large number of commands would be:

MParallel.exe [options] --input=commands.txt

...where the text file commands.txt contains all your commands to execute – one command per line.

Also I highly recommend you familiarize yourself with the --pattern option 😉

---

If you generate your commands in Java, then you may even do:

java.exe -jar my_command_generator.jar | MParallel.exe --stdin

...where your Java program my_command_generator.jar could look something like this:

public static void main(String [] args) {
    System.out.println("command 1");
    System.out.println("command 2");
    [...]
}

(You get the idea ☺️)

[jgmeyerucsd]

Awesome, thanks for the suggestion! Seems obvious to use a command txt file I didn't realize you had that support. Thank you!

[lordmulder]

Yes, using a text file should be way better for a long list of commands.

But you can even pipe the commands directly from your Java program, so no temporary text file needed.

---

Also, with a pattern you can greatly simplify "similar" commands:

If, for example, you want to run many commands like program.exe -i "input_xxx" -o "output.xxx".

You can simply do:

MParallel.exe --pattern="program.exe -i \"{{0}}\" -o \"{{1}}\"" --input="file_list.txt"

...where file_list.txt contains the parameters to be filled in, e.g. the path of the input/output file.

For example, it could look something like this:

input_001.foo output_001.bar
input_002.foo output_002.bar
input_003.foo output_003.bar
[...]