Use SSIM as progress without normalization
Opened this issue · 3 comments
A followup to #48, we should follow @pahammad's advice and do what visualmetrics does, using SSIM as the progress once the first paint has occurred and do not normalize.
The normalization causes problems on sites like www.chromestatus.com where a white page has a SSIM of 60 (!!!) with the target. This greatly exaggerates the small differences in SSIM later in the page load since the remaining 40% is remapped to 100%.
PSI is ~2600 under the flawed approach when its realistically more like ~1600.
Not normalizing at all also isn't ideal either, the PSI of twoframes-blank_content.json gets cut in half because the first paint of white with a very thin black border has 83 SSIM with the content :/
This example you showed reminds me of a video Pat Meenan sent me a while ago, where the webpage just had the text "Hello World!" on a blank page, right after the firstPaint. Such a webpage loading video's PSI should be very close to firstPaint (plus some delta); and this is the correct behavior to expect in my opinion. firstPaint can be replaced with startRender (or firstVisualChange) since these metrics tend to be very close; just different terms used in different repositories.
The problem here is that in this case, the text is not right after firstPaint, it's substantially farther out, so PSI in this case being very close to firstPaint (which from the user's point of view is indistinguishable from a blank white page) is not a very good outcome. All of the progress from the user's perspective here is when the first text loads, not from a blank white frame with an imperceptible border being painted.