lucaslorentz/minicover

Performance

MCFHTAGENTS opened this issue · 10 comments

MCFHTAGENTS commented

Hi there
i am generally finding this tool quite slow. I have a relatively small website and the total process of running the reset, build, instrument, xmlreport and conversion is taking approx. 12 hours.

I suspect, but haven't yet proven that my anti-virus system will be the culprit but I note that I have of the order of 1.2 million files in the content-hits directory and generally I have seen poor performance across Windows when there are large number of files in a directory.

Generally, it would seem to me better if the content-hits were stored in a database. Either a relational one or a file based one. This is a suggestion for such an enhancement.

Thanks
Mark

lucaslorentz commented

Interesting, can you please try to reference your test sources as well? Using --tests parameter.

Let me know if that changes the performance

lucaslorentz commented

I will try to fix that in the next days, it is a known issue but I expected it to be slightly better now.

MCFHTAGENTS commented

Hi - I am using the --tests parameter :(. Looking at the code a few observations:

  • GetFiles is Slower that EnumerateFiles when running the Reset command
    See here
    and here
    I will switch off the AV when my current run has completed and I can try it again
    To give some data executing 176 tests took 3.2 hours.
    Am sure putting the data in a database would help!

Will update once I have more info.

Thanks
Mark

MCFHTAGENTS commented

ok - I think I can see what is happening here - my machines are all using file level backup solutions. We are shovelling 1m files across it and I can't restrict it by location so each of those files is moving in multiple places. Hmmmmm

lucaslorentz commented

I recently switched to an approach that creates too many small files.
That was clearly a bad decision, files are too small and the overhead of creating them doesn't worth it.
I will switch back to less but bigger files.

lucaslorentz commented

I think this will improve your case: 927a4b2

I will package a version now.

lucaslorentz commented

Please, try again with version 3.0.4. It will be published soon.

MCFHTAGENTS commented

Thank you - working through now. Will up date in due course
Thanks
Mark

MCFHTAGENTS commented

Thank you - confimed fixed. - down to mere minutes