Fuzzy image comparison a good idea?
lars-erik opened this issue · 1 comments
Hi y'all!
TLDR; See bottom paragraph.
I spent the last three days building something I have no idea whether was smart or not. 👼
The background is that we have a fresh new designer who is going through all the regression fault experiences we've all gone through. Change something here, something else looks different there.
So I thought I'd see if we could implement a safeguard mechanism using Selenium and ApprovalTests.
Why not just take screenshots of stable states and compare archetype pages to approved screenshots?
Granted it gives us the responsibility of updating acceptance criterias, but surely it should act as a nice safety net
for a junior developer learning SASS?
So I wrote a couple of test cases that verified screenshots using Approvals.VerifyBinaryFile. It seemed to work fine,
but when I ran the tests on Azure Pipelines, the test blew up. I managed to attach the received file to the test output
and discovered that the build server rendered the Oswald font from Google differently than our dev. boxes.
I must admit I have no idea (although fiddled a LOT) whether it's possible to make chromium output consistent
images on all boxes running the tests, but I gave up and figured there must be a way to allow that difference, but
fail on any excessive difference. So I went ahead and wrote an approver that does a CIE76 comparison of the
received and approved image. If the ratio of noticably different pixels are above a certain limit the approval will fail.
If it's below the limit, the screenshot will be approved, even if it's not entirely similar.
In addition to just doing the fuzzy comparison, there's a reporter that will attach a color diff image, a grayscale CIE76 diff representation, a JSON file with some statistics about the differences, as well as the screenshot and approval file.
There's also a (for now) disabled mechanism to mutate the image with for example a box or gaussian blur before the comparison is done.
The bottom line is that I'm aching for input / feedback here. Is this totally stupid? Is the wheel already invented? Should I remove "ApprovalTests" from the repo / namespaces due to copyrights? Would you want to merge it into core? What stupid mistakes did I make? If not, isn't this totally awesome? 😆
The code is over here for now with some simple tests showing the gist of it:
https://github.com/lars-erik/approvaltests.imageapproval
Hope to hear from you!
Lars-Erik
This project is not being actively maintained. Instead consider using Verify. See Migrating from ApprovalTests for more information.