Comparison process ends early

[rossjrw]

I'm seeing a frequent issue where the comparison process simply ends mid-way through comparing images. The comparison does not make it through all the images, does not get as far as executing openReport, and does not generate the html_report in the first place (but does generate diff images for comparisons that it was able to run before stopping. Despite all this, it exits with code 0.

I believe this happens when asyncCompareLimit is too high and the process runs out of memory. This is corroborated by #1091 (closed without resolution), #1182 and #1240, which I believe to be the same issue.

#1091 was remedied for the reporter by increasing the RAM on their system, but principally, Backstop should throw an error when this happens, so that it can at least be detected. Even better, it could handle this event gracefully and continue to compare images with a lower asyncCompareLimit.

In our process we are detecting this state by checking for the html_report dir after Backstop has finished. Prior to adding this check though, we were at a loss as to why the HTML report was present only sometimes.

[ivnish]

Same problem. I have a lot of RAM, but report doesn't generate. I decrease asyncCompareLimit and it works. Thanks!