Parallel doesn't play well with filenames containing a quote

[WyohKnott]

Hello, it's me again with my filenames problems

I'm gonna take an example to explain the situation.

Say we have two folders with files to move.

mkdir folder1 folder2
touch "folder1/Let'sgo.txt" "folder1/Let'sgo-q1.txt" "folder1/Let'sgo-q2.txt" "folder1/Let'sgo-q3.txt"
export file="Let'sgo"

I can move a file containing a quote with mv without any problem:

mv "folder1/${file}.txt" "folder2/."

However if I try to use parallel to do the same thing, I get an error:

seq 1 3 | parallel mv "folder1/${file}-q{}.txt" "folder2/."

parallel: reading inputs from standard input
mv: missing destination file operand after 'folder1/Letsgo-q1.txt folder2/.'
Try 'mv --help' for more information.
mv: missing destination file operand after 'folder1/Letsgo-q2.txt folder2/.'
Try 'mv --help' for more information.
mv: missing destination file operand after 'folder1/Letsgo-q3.txt folder2/.'
Try 'mv --help' for more information.

It seems parallel remove the quote from my filename or misinterpret it, despite it being inside an enclosed variable.

If I escape the quote, it works:

seq 1 3 | parallel mv "folder1/Let\'sgo-q{}.txt" "folder2/."

but it's not really a workable solution because I feed filenames as a variable inside a bigger script. I think there shouldn't be any difference of behaviour between a non-parallelized function and a parallelized one.

[mmstick]

Basically, your shell expands the variable before it passes it to the program, so there's no way to tell what is or isn't inside a variable. Although this is just a general parsing problem that should be simple to fix.

[mmstick]

I'll have the fix released sometime tomorrow. Basically, I'll be escaping input parameters by default, and implementing the -q / --quote parameter which will escape the command argument. With the input you're trying, this will generate a parsing error without --quote due to having a non-terminated quote, but will work as you want with --quote enabled.

[mmstick]

This commit should resolve the issue. If the problems are fixed, you can close this issue and I'll make a new release.

Try it out like so:

seq 1 3 | parallel -q mv -v "folder1/${file}-q{}.txt" "folder2/."

There is also a custom error message if you don't use quoting that will tip you to use the --quote parameter to escape special characters.

[WyohKnott]

It doesn't seem to work, the {} is not replaced with the input anymore when using -q.

seq 1 3 | parallel -q mv "folder1/${file}-q{}.txt" "folder2/."
parallel: reading inputs from standard input
mv: target '1' is not a directory
mv: target '2' is not a directory
mv: target '3' is not a directory

If I use echo to see what happens, parallel thinks there is no {} in my command so it appends the variables at the end:

seq 1 3 | parallel -q echo "folder1/${file}-q{}.txt" "folder2/."
parallel: reading inputs from standard input
folder1/Let'sgo-q{}.txt folder2/. 1
folder1/Let'sgo-q{}.txt folder2/. 2
folder1/Let'sgo-q{}.txt folder2/. 3

[WyohKnott]

I've looked at the diff you've made, and it seems it's the last commit 6686eb7 which messes things up. After doing a checkout of f4f1e23 and compiling it, it works as intended.

[mmstick]

It should be fixed now. The braces are no longer escaped.

[WyohKnott]

It is working fine now. Thank you.