paulirish/speedline

White frame perceptual progress can be significantly greater than 0

Closed this issue · 3 comments

patrickhulce commented

The problem now that we're using real SSIM is that getPerceptualProgress normalizes the value relative to the global min instead of the SSIM of the white frame with the target frame. This creates some very unintuitive histograms based on the fact that a page with a white background can be quite structurally similar to a white frame.

A recent trace of CNN highlights this issue. Their home page loads content with header, image, and text before an ad pops in above that pushes all the content down. The frame before the ad has a structural similarity with the target of ~18% while empty white has a structural similarity of ~30% which results in the following histogram.
image

This artificially deflates the perceptual speed index for credit before first paint, and does not appropriately punish the site for progressively rendering toward a target that is abruptly changed. The histogram should instead look like the following if normalized to the first frame:
image

Before Ad After Ad
image image
pahammad commented

We explicitly addressed this issue (in VisualMetrics repository) earlier by starting the pairwise frame comparison from firstPaint frame instead of time=0 (white) frame. The modified PSI equation looks like: PSI = firstPaint + sum (from firstPaint to end) {Perceptual_pairwise_FrameDifference}. This means you don't have to compute pairwise frame differences upto firstPaint. This may make the compute faster as well. In this formula, Perceptual_pairwise_FrameDifference is computed using SSIM.