hideSelectors not working for reference, but for test (puppeteer)

[JPustkuchen]

I'm using hideSelectors on ".slick" slideshows as timing may otherwise cause differences.

For "test" everything works fine and as expected, but for "reference" the hideSelectors don't seem to work:

(The head should be hidden in both, but only test is correct)

This is my backstop.json:

{
    "viewports": [
        {
            "name": "large",
            "width": 1280,
            "height": 920
        },
        {
            "label": "medium",
            "width": 1024,
            "height": 768
        },
        {
            "label": "small",
            "width": 640,
            "height": 920
        }
    ],
    "paths": {
        "bitmaps_reference": "backstop_data/bitmaps_reference",
        "bitmaps_test": "backstop_data/bitmaps_test",
        "engine_scripts": "backstop_data/engine_scripts",
        "html_report": "backstop_data/html_report",
        "ci_report": "backstop_data/ci_report"
    },
    "report": [
        "browser"
    ],
    "engine": "puppeteer",
    "engineOptions": {
        "args": [
            "--no-sandbox"
        ]
    },
    "asyncCaptureLimit": 5,
    "asyncCompareLimit": 50,
    "debug": false,
    "debugWindow": false,
    "misMatchThreshold": 2,
    "scenarios": [
        {
            "label": "/wohnungsbau",
            "url": "https://new.example.com/example",
            "referenceUrl": "https://www.example.com/example",
            "selectors": [
                "document"
            ],
            "hideSelectors": [
                "iframe",
                ".slick",
                ".blazy",
                ".captcha",
                "#cookiesjsr"
            ]
        },
...

What am I doing wrong? I already deleted the entire backstop_data folder and run it again. I'm having the same issue on multiple different projects with similar configuration.

[edvardsniedre]

Have you tried running backstop reference and then backstop test?

[firelf15]

Is it always the same, which command produces the correctly hidden selector result and which has the errant result? This kind of thing happens to me often enough that it's kind of "broken is normal." If I recall correctly, it's not always wrong on the same command; sometimes reference is correct, sometimes test. I'm looking at solutions for LazyLoading [my current suspect].
Update: The trick here https://github.com/garris/BackstopJS/issues/1330#issuecomment-890512892 has helped my situation tremendously with the lazy loading; when combined with onReady functions that change all the loading="lazy" into loading="eager" and wait for fonts to load.

[JPustkuchen]

Thanks @firelf15 that was helpful in other situations, but didn't solve the general issue for me.
@edvardsniedre Yes that was the way I did that. With the results above.

So for me it still seems the hideSelectors doesn't apply on reference. But it should, shouldn't it?

[calebtr-metro]

I have observed the same behavior with Puppeteer and Backstop 6.3.3, trying to hide a pesky slideshow in page-level snapshots.

Using removeSelectors instead of hideSelectors resolved the issue for me.

I also tried lowering the asyncCaptureLimit parameter, and that reduced the problem. Setting it to 1 resolved it completely. If this issue is caused in part by a lack of system resources, that could explain why this behavior would be inconsistent for developers.

Also, since hideSelectors uses the CSS visibility property, it could be that some slideshows' use of that same property are interfering with the behavior we are expecting from Backstop and Puppeteer.

In the end, obliterating the slideshow was most effective.