Image diff tools is nice, but confusing
Opened this issue · 1 comments
After running the automated test suite in vcs using the run_tests.py --html approach, I'm looking at the failures reports using the image diff tool. In the screenshot below, which is the known-good baseline image and which is the output from the failed test? You might think, as I did initially, that the green outline indicates the known-good baseline and the red outline indicates the output of the failed test. But in fact it seems that this is backwards. After quite awhile of scratching my head, digging into code, and ending up at this repo, I believe that the green outline is the erroneous test output, while the red outline is the known good baseline.
Either way, could there be some indication in the web page which is which? Like a legend indicating that "green = bad, red = good"? Or since the image-diff tool itself came from an npm library whose api more closely reflects a git diff tool (so red is "out/before", while green is "in/after" some change), perhaps we could just switch the order of the arguments so that green means known-good and red means test failure image?
If people like the out/before vs. in/after semantics better, then a legend to alert new users might be enough. If people prefer the known-good vs test-failure semantics, then maybe we switch the arguments to the image-diff method.
@scottwittenburg good point, if i ever get time I'll look into this. In the mean time, whatever picture shows up first is the test picure, slider reveals the bad one.