Mismatchthreshold not consistently working

[aschoelling]

I use a sitemap.xml based Gitlab CI/CD Job, and I regularly get results tests declared ok with extreme differences between MisMatchThreshold (5% currently) and reported difference (I've seen values significantly higher than 50% being declared OK.

Configuration File for TEST:

const basicConfig = require("./basic");
const ONE_SECONDS_IN_MS = 1000;
const scenarios = [];
const viewports = [];
const bviewports= ["desktop"];

// Creates the list of scenarios (urls to screenshot)
basicConfig.relativeUrls.map(relativeUrl => {
    scenarios.push({
        label: relativeUrl,
        url: `${basicConfig.testUrl}${relativeUrl}`,
        delay: ONE_SECONDS_IN_MS,
        requireSameDimensions: false,
        misMatchThreshold: process.env.THRESHOLD, // Number(process.env.THRESHOLD) didn't work, either
        hideSelectors: ['.cookieconsent'],
        // Could be used to hide (and therefore ignore) youtube videos
    });
});

bviewports.map(viewport => {
    switch(viewport){
        case "phone":
            pushViewport(viewport, 320, 480);
            break;
        case "tablet":
            pushViewport(viewport, 1024, 768);
            break;
        case "desktop":
            pushViewport(viewport, 1280, 1024);
            break;
    }
});

function pushViewport(viewport, width, height) {
    viewports.push({ name: viewport, width, height });
}

module.exports = {
    id: basicConfig.projectId,
    viewports,
    scenarios,
    paths: {
        bitmaps_reference: 'backstop_data/bitmaps_reference',
        bitmaps_test: 'backstop_data/bitmaps_test'
    },
    report: ["browser", "CI"],
    engine: "puppeteer",
    engineOptions: {
        args: ["--no-sandbox"]
    },
    asyncCaptureLimit: 5,
    asyncCompareLimit: 50,
};

The variables are filled in basic.js from GITLAB variables:

const baseUrl = process.env.REFHOST;
const testUrl = process.env.TESTHOST;
const projectId = process.env.PID;

const url = require('url');
const urls = require('./urls.json'); // Contains an array of url strings
const relativeUrls = urls.map(absUrl => {
    return url.parse(absUrl, false, true).pathname;
});

module.exports = {
    baseUrl,
    testUrl,
    projectId,
    relativeUrls,
};

urls.json is created by Gitlab Job and works, I can see all pages in the generated report

[aschoelling]

I just did some more tests, and I think the error is somehow related to Gitlab. Rewrote the tests (minor changes only) to run standalone from a BASH-Script, the results are OK, no passing page with more than 5% difference anymore. I think this issue can be closed for now, unless someone has an idea on what is happening.

[garris]

Thanks for doing more research and working around the issue.

Maybe you could post your solution so others can learn from your experience.

Cheers.

[aschoelling]

I'll see to it when my current workload gives me some room for it, for now I'm content with the knowledge that it works locally. Will update this issue when I got new information.