fingerprint injection makes browsers detectable as headless
Opened this issue · 20 comments
Describe the bug
Simply injecting fingerprint-suite (using newInjectedContext or newInjectedPage), with or without options, for headless or headfull browser, makes the browser detected as headless in : https://arh.antoinevastel.com/bots/areyouheadless.
- headless = detected
- headless + stealth = undetected
- headless + fingerprint-suite = detected !!
- headless + stealth + fingerprint-suite = detected !!
- headfull = undetected
- headfull + stealth = undetected
- headfull + fingerprint-suite = detected !!
- headfull + stealth + fingerprint-suite = detected !!
Every time fingerprint suite is used, even without any option, it makes the browser detectable.
To Reproduce
const { chromium: playwright } = require('playwright-extra')
const { newInjectedContext } = require('fingerprint-injector');
playwright.launch({ headless: false }).then(test)
async function test(browser) {
const context = await newInjectedContext(browser, {}); // DETECTED
// const context = await brower.newContext(); // UNDETECTED
const page = await context.newPage();
await page.goto('https://arh.antoinevastel.com/bots/areyouheadless');
await page.screenshot({ path: 'detected.png', fullPage: true });
await browser.close()
}Expected behavior
Injecting the fingerprint-suite shouldn't make the browser be detected as headless.
System information:
- OS: Arch Linux x86_64 - 6.3.2-arch1-1
- Node.js version: v16.20.0
It seems like using the "chrome" browser ( browsers: ["chrome"] ) is the only one triggering the detection.
// headless mode
Device: desktop | Os: windows | Browser: chrome | Status: You are Chrome headless
Device: desktop | Os: windows | Browser: firefox | Status: You are not Chrome headless
Device: desktop | Os: windows | Browser: edge | Status: You are not Chrome headless
Device: desktop | Os: macos | Browser: chrome | Status: You are Chrome headless
Device: desktop | Os: macos | Browser: firefox | Status: You are not Chrome headless
Device: desktop | Os: macos | Browser: edge | Status: You are not Chrome headless
Device: desktop | Os: macos | Browser: safari | Status: You are not Chrome headless
Device: desktop | Os: linux | Browser: chrome | Status: You are Chrome headless
Device: desktop | Os: linux | Browser: firefox | Status: You are not Chrome headless
Device: desktop | Os: linux | Browser: edge | Status: You are not Chrome headless
Device: mobile | Os: android | Browser: chrome | Status: You are not Chrome headless
Device: mobile | Os: android | Browser: firefox | Status: You are not Chrome headless
Device: mobile | Os: android | Browser: edge | Status: You are not Chrome headless
Device: mobile | Os: ios | Browser: edge | Status: You are not Chrome headless
Device: mobile | Os: ios | Browser: safari | Status: You are not Chrome headless
// headfull mode
Device: desktop | Os: windows | Browser: chrome | Status: You are Chrome headless
Device: desktop | Os: windows | Browser: firefox | Status: You are not Chrome headless
Device: desktop | Os: windows | Browser: edge | Status: You are not Chrome headless
Device: desktop | Os: macos | Browser: chrome | Status: You are Chrome headless
Device: desktop | Os: macos | Browser: firefox | Status: You are not Chrome headless
Device: desktop | Os: macos | Browser: edge | Status: You are not Chrome headless
Device: desktop | Os: macos | Browser: safari | Status: You are not Chrome headless
Device: desktop | Os: linux | Browser: chrome | Status: You are Chrome headless
Device: desktop | Os: linux | Browser: firefox | Status: You are not Chrome headless
Device: desktop | Os: linux | Browser: edge | Status: You are not Chrome headless
Device: mobile | Os: android | Browser: chrome | Status: You are Chrome headless
Device: mobile | Os: android | Browser: firefox | Status: You are not Chrome headless
Device: mobile | Os: android | Browser: edge | Status: You are not Chrome headless
Device: mobile | Os: ios | Browser: edge | Status: You are not Chrome headless
Device: mobile | Os: ios | Browser: safari | Status: You are not Chrome headless
const { chromium } = require("playwright");
const { FingerprintInjector } = require("fingerprint-injector");
const { FingerprintGenerator } = require("fingerprint-generator");
async function runBrowser(device, os, browserType) {
const browser = await chromium.launch({
headless: false
});
// chrome, firefox, edge, safari
const { fingerprint, headers } = await new FingerprintGenerator().getFingerprint({
devices: [device],
operatingSystems: [os],
browsers: [browserType]
});
const context = await browser.newContext({
userAgent: fingerprint.navigator.userAgent,
colorScheme: 'dark',
viewport: {
width: fingerprint.screen.width,
height: fingerprint.screen.height,
},
extraHTTPHeaders: {
'accept-language': headers['accept-language'],
},
});
await new FingerprintInjector().attachFingerprintToPlaywright(context, { fingerprint, headers });
const page = await context.newPage();
await page.goto("https://arh.antoinevastel.com/bots/areyouheadless", { waitUntil: "load"});
const value = await page.evaluate(() => document.querySelector('#res p').textContent);
console.log(`Device: ${device} | Os: ${os} | Browser: ${browserType} | Status: ${value}`);
await browser.close();
}@steinpigs thanks for this test!
any takers ?
Sorry for the delay, I actually started looking into this last week - but unfortunately didn't come to any conclusion. The 'Are you headless' website seems to utilize some kind of ML-like regression, where it collects the browser fingerprint and then sends it to the server, which decides whether the fingerprint is valid or not.
Since this is not an actual bot-protection service, the priority on this is a bit lower, but I'll definitely continue looking into this. Thanks for your patience! :)
No problem, I understand. Thank you for the update. However, I wanted to mention that the injected evasions in the fingerprint-injector compromise the anonymity of the fingerprint, rendering it unusable. This is an important factor to consider.
I'm not sure if this works here, but you should find a way to use 'new' in headless
playwright.launch({ headless: new }).then(test)
I'm not sure if this works here, but you should find a way to use 'new' in headless
playwright.launch({ headless: new }).then(test)
Thanks for the inputs, but that's not the issue here.
Plus, the headless new option is only for puppeteer and not playwright : https://github.com/microsoft/playwright/blob/9bca9f1b4ff3478c20a526c8f8b41ab8ab9be6e6/packages/playwright-core/src/client/types.ts#L105
Debugged this, It is getting flagged due to webDriver property in the fingerprint being set to true.
The https://arh.antoinevastel.com/bots/areyouheadless plugin has a check
https://github.com/antoinevastel/fpscanner/blob/master/src/fpScanner.js#L119
@abhisheksurve45 So why did not fingerprint-suite set webDirver to false?
fingerprint-injector/fingerprint-injector.js
await page.setExtraHTTPHeaders(this.onlyInjectableHeaders(headers, browserVersion));
without extraheader will pass the detection.
fingerprint-injector/fingerprint-injector.js await page.setExtraHTTPHeaders(this.onlyInjectableHeaders(headers, browserVersion));
without extraheader will pass the detection.
Removing accept-language will fix this issue:
fingerprint-injector/fingerprint-injector.js
await page.setExtraHTTPHeaders(this.onlyInjectableHeaders(headers, browserVersion));
++ delete extraHeaders['accept-language'];
Can confirm @tenkuken patch works! I added accept-language to the requestHeaders array (which is an array of headers to be filtered out) and my tests with areyouheadless works both with headless: old and headless: new on puppeteer
@barjin Are you planning a fix or customizable header settings that don't involve forking this?
The page.on("request") approach is one way of editing the header as well. But I think in the options we should incorporate something like:
//conf:
ignoreHeaders: ['accept-language']
That way:
[...requestHeaders, ...ignoreHeaders].forEach((header) => {
delete filteredHeaders[header];
});
@barjin Isn't the guy from that page https://arh.antoinevastel.com/bots/areyouheadless working for Datadome a bot-detection company. Which would make this maybe a priority?
@iwaduarte Make it your priority then :) PRs are welcome.
Regarding the fix - I would rather not add another option in the already pretty granular options object. The other problem is that by removing the accept-language header, the fingerprint becomes less consistent (as puppeteer/playwright will probably substitute the header with a default, while still leaving the navigator.languages API injected).
A proper solution would include hiding the headlessness without compromising the other parts of the injection - which is the first (and only) rule when introducing new features into this library.
As I said, PRs (preferably with proper research/tests) are welcome. Thanks!
@barjin I think you have to be more specific here. I could indeed drop a PR for the repo but "would include hiding the headlessness without compromising the other parts of the injection" does not give much to work with. How would you define compromise?
And also if you could give a tip of what code you advise be even better :)
💎 cloudgakkai is offering a $30 bounty for this issue
👉 Got a pull request resolving this? Claim the bounty by adding @algora-pbc /claim #178 in the PR body and joining algora.io
Hello everyone, I love this lib. Therefore, I'm supporting by giving bounties as CloudGakkai here.
The main goal is I want to make fingerprint-suite undetectable on puppeteer
Can confirm with rebrowser-patches. Suno with hCaptcha easily detects fingerprint-suite and generates a 'Fake app' song even if the hCaptcha is successfully solved. Happens even with the 'accept-language' patch. Without the suite, hCaptcha doesn't detect anything bad, but it is constantly requesting to solve CAPTCHAs, probably because of bad fingerprints, which I'm trying to solve. Welp, we'll still have to constantly solve CAPTCHAs then.
Make it your priority then :) PRs are welcome.
Sorry if I'm being too harsh, but what's the point of this project then? Is it a DIY guide? This problem should have already been solved a year ago because this is a constantly updated set of already built tools for making a good fingerprint that will at least try to avoid CAPTCHAs, not invite them.
Hi everyone,
Apologies for the long wait and for pinging the thread - this discussion has grown quite a bit over time.
To address why other tools might seem more effective: most of them focus on directly patching the obvious signs of automation. fingerprint-suite, however, takes a different approach by randomizing values like user-agent, accept-language, screen size, and more. This strategy is designed to make large-scale scraping resemble regular traffic, rather than simply hiding automation markers. While this approach is less predictable, it aligns more closely with real user behavior. I'm not saying this is the best approach here - maybe using similar tricks as the other libraries are doing would be the right way forward.
As for the age of this issue, we understand the frustration. Unfortunately, we cannot dedicate reasonable manhours to this right now due to other project commitments. Solving this issue correctly would require extensive testing and research. While we’re unable to prioritize this internally, contributions from the community are always welcome.
Thank you for your patience and understanding.