Problems with temp files when running multiple instances of precomp

[M-Gonzalo]

I know, I know, that's a horrible description ^

Only I don't know how else to put it. Since the last commit (at least I didn't notice it before), precomp crashes or exits before completion when there are more than one instance of the program working in parallel. It seems to me that the problem is related to some kind of racing condition with the temp files. Sometimes i get an error like this:

29.13% / ERROR: Access denied for ~temp000000092.dat

It doesn't happen when the same file is processed alone.
I don't believe also that it has anything to do with my hard drive because the issue appears when I'm working from a ram drive too.

[schnaader]

@M-Gonzalo: The commit should fix this, please check if it works for you. Also note that the LZMA part of precomp as well as the latest preflate changes both use multithreading, so it might not be faster spawning multiple instances anymore. For example, for me it took 1 min 30 s to compress silesia.zip using two instances, but only 55 s when using only one instance.

[M-Gonzalo]

@M-Gonzalo: The commit should fix this, please check if it works for you. Also note that the LZMA part of precomp as well as the latest preflate changes both use multithreading, so it might not be faster spawning multiple instances anymore. For example, for me it took 1 min 30 s to compress silesia.zip using two instances, but only 55 s when using only one instance.

It seems to work so far.
And yes, I'm aware of that. Still, I usually use precomp in -cn mode, leaving the compression to some more advanced external engine, so I don't really use the internal lzma that much.

There are several cases where parallel execution gives nice speed ups. It's far from ideal, it doesn't scale linearly, but it's the best we have for now. See a few examples of precomp running 1, 2 and 4 processes in parallel, but using no bult-in compression, on a 4 cpu computer:

non-zlib deflate:

1	3m20,154s	100%

2	1m18,074s	39%

4	1m1,495s	30,5%



zlib deflate(*):

1	13m47,817s	100%

2	7m30,754s	54%

4	6m30,330s	47%


jpg:

1	9m16,166s	100%

2	4m17,690s	46%

4	3m19,877s	36%


mp3:

1	18m0,386s	100%

2	9m15,848s	51%

4	7m13,194s	40%

The test sets are of different sizes. The timing is measured running 5 times an script on the folder of the test set, containing a rather large number of relatively small files.
The script is run from a ram drive to minimize the impact of the temp files.
You can expect similar numbers for gif too.
(*)This is a bunch of small .jar files from two versions of zpaqgui, so there is a lot of overhead due to the presence of thousands of temp files

[M-Gonzalo]

Now, about preflate multithreading, I'm not that sure it is really working or if it helps at all. Maybe with very large streams only? I mean, look at the improvements in the speed after adding a second process...
It could be a flaw in the measurement, though (I didn't really expect the timing to go below 50%). Maybe you can check it out?