puppeteer .waitForSelector causes error (evalmachine.<anonymous>

Question

puppeteer .waitForSelector causes error (evalmachine.<anonymous>

harimayco opened this issue 2 years ago · 8 comments

Hi,

I'm facing this issue when I tried to run puppeteer with bytenode. anyone can help?
Script

async doLogin() {

    console.log(this.browser.page.url());

    const loginPage = this.browser.page.url().includes(this.loginEndpoint);

    //if stuck in login page, maybe email or password is wrong
    if (loginPage) {
      console.log('1');
      await this.browser.page.waitForSelector('#identifierId', { visible: true })
      console.log('2');
      await this.browser.page.type('#identifierId', this.browser.browserOptions.account.email, { delay: 0 })
      console.log('3');
      await this.browser.page.keyboard.press('Enter')

here is the debug info

User Trying to login
https://accounts.google.com/signin/v2/identifier?flowName=GlifWebSignIn&flowEntry=ServiceLogin
1
Error: Login Gagal
    at BotGoogleLogin.<anonymous> (evalmachine.<anonymous>:1:2629581)
    at step (evalmachine.<anonymous>:1:2628046)
    at Object.throw (evalmachine.<anonymous>:1:2626823)
    at rejected (evalmachine.<anonymous>:1:2626164)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[3676:0409/222919.180:ERROR:gpu_init.cc(457)] Passthrough is not supported, GL is disabled, ANGLE is

However If I disable Bytenode Compiler, it runs smoothly withour error.
disabled BytenodeWebpackPlugin:

called Login Funct
User Trying to login
https://accounts.google.com/signin/v2/identifier?flowName=GlifWebSignIn&flowEntry=ServiceLogin
1
2
3
login sukses

OS Windows 11
puppeteer-core: 13.5.2
bytenode: 1.3.5
bytenode-webpack-plugin: 1.2.5
@electron-forge/plugin-webpack: ^6.0.0-beta.54
@electron-forge/cli: ^6.0.0-beta.61
ts-loader: ^8.3.0
typescript: ^4.5.3

is there something wrong with my project setup ?

Answer 1 · 2022-04-09T17:06:45.000Z

Can you create a minimal complete example that can be run simply using npm start?

Answer 2 · 2022-04-09T17:42:09.000Z

Can you create a minimal complete example that can be run simply using npm start?

ofcourse sir,
here it is https://github.com/harimayco/bytenode-electron-puppeteer-error

you can set chrome executable chrome path in main.ts regarding your OS
Many Thanks!

Answer 3 · 2022-04-09T19:44:10.000Z

First, you should have logged this error in this line:
https://github.com/harimayco/bytenode-electron-puppeteer-error/blob/991c16c0f53f4852367e451603e7571e0da1dce2/src/googleLogin.ts#L24

I logged it (along with the __filename), and here is what I got:

Error: Passed function is not well-serializable!
    at ExecutionContext._evaluateInternal (evalmachine.<anonymous>:1:547522)
    at ExecutionContext.evaluateHandle (evalmachine.<anonymous>:1:545423)
    at WaitTask.rerun (evalmachine.<anonymous>:1:470901)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
/home/osama/Desktop/Dev/bytenode-electron-puppeteer-error/.webpack/main/main.compiled.jsc

This is strange for two reasons:

(1) the .waitForSelector function works properly with bytenode. If you try the example given here, it works with bytenode.

(2) the .waitForSelector does not have any function to be serialized. If this was the issue, then my workaround here would have solved it.

I'm still investigating the issue. My initial guess is that webpack does something which causes this strange behavior.

Answer 4 · 2022-04-09T20:15:19.000Z

So, I have simplified your example significantly by removing everything except bytenode and puppeteer, and it worked just fine with and without bytenode.

Here is the simplified version of your work:

package.json:

{
  "name": "issue-179",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@seald-io/nedb": "^3.0.0",
    "@types/puppeteer": "^5.4.5",
    "@types/puppeteer-core": "^5.4.0",
    "bytenode": "^1.3.5",
    "electron": "^16.0.4",
    "electron-bytenode-webpack-plugin": "^1.1.2",
    "electron-squirrel-startup": "^1.0.0",
    "puppeteer-core": "^13.5.2",
    "puppeteer-extra": "^3.2.3",
    "puppeteer-extra-plugin-stealth": "^2.9.0",
    "unlazy-loader": "^0.1.3"
  }
}

index.js:

'use strict';

require('bytenode');
require('./main.jsc');

main.js:

'use strict';

const BotGoogleLogin = require('./googleLogin.jsc');
const BotBrowserService = require('./browser.jsc');

const acc = {
  _id: '_3242342',
  email: 'test@gmail.com',
  password: 'test',
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36 Edg/100.0.1185.29'
}
start(acc);

async function start(account) {

  //const chromeExecutablePath: string = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'; // for mac os User
  const chromeExecutablePath = '/usr/bin/google-chrome';

  const browserOptions = {
    headless: false,
    account: account,

  }

  const browser = new BotBrowserService(browserOptions);
  browser.executablePath = chromeExecutablePath;

  await browser.openBrowser();

  const googleLogin = new BotGoogleLogin(browser);
  try {
    await googleLogin.login();
    console.log('login sukses');
    // const issuu = new IssuuBotService(browser, );
    // await issuu.run();
    //browser.browser.close();
  } catch (err) {
    console.log(err);
  } finally {
    // browser.browser.close();
  }
}

googleLogin.js:

'use strict';

module.exports = class BotGoogleLogin {

  browser;
  loginEndpoint = 'https://accounts.google.com/signin/v2/identifier';
  successEndpoint = 'myaccount.google.com';

  constructor(browser) {
    this.browser = browser;
  }

  async login() {
    try {
      const isLoggedIn = await this.checkLogin();
      console.log('called Login Funct');
      if (isLoggedIn) {
        console.log('User Already Logged In');
        return true;
      }
      console.log('User Trying to login');
      return await this.doLogin();
    } catch (err) {
      console.log(err);
      console.log(__filename);
      throw new Error(`Login Failed`)
    }
  }

  async doLogin() {

    console.log(this.browser.page.url());

    const loginPage = this.browser.page.url().includes(this.loginEndpoint);

    //if stuck in login page, maybe email or password is wrong
    if (loginPage) {
      console.log('1');
      await this.browser.page.waitForSelector('#identifierId', { visible: true })
      console.log('2');
      await this.browser.page.type('#identifierId', this.browser.browserOptions.account.email, { delay: 0 })
      console.log('3');
      await this.browser.page.keyboard.press('Enter')
      // await page.waitForTimeout(1000);

      await this.browser.page.waitForSelector('#password input[type="password"]', { visible: true })
      await this.browser.page.type('#password input[type="password"]', this.browser.browserOptions.account.password, { delay: 0 })
      // await newPage.click('#passwordNext');
      await this.browser.page.keyboard.press('Enter')
      await this.browser.page.waitForTimeout(1000)
      await this.browser.page.waitForNavigation()
    }

    // else we wait until my account page shows
    await this.browser.page.waitForRequest(request => {
      return request.url().includes(this.successEndpoint)
    })

    return Promise.resolve(this.browser.page)
  }

  async checkLogin() {
    await this.browser.page.goto(this.loginEndpoint, { waitUntil: 'networkidle2', timeout: 0 })
    let isLoggedIn = this.browser.page.url().includes(this.successEndpoint);

    return isLoggedIn;
  }

}

browser.js:

'use strict';

const { app } = require('electron');
const puppeteerVanilla = require('puppeteer-extra');
const { Browser, Page } = puppeteerVanilla;

const { addExtra } = require('puppeteer-extra');

const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const ChromeAppePlugin = require('puppeteer-extra-plugin-stealth/evasions/chrome.app');
const DefaultArgsPlugin = require('puppeteer-extra-plugin-stealth/evasions/defaultArgs');
const ChromeRuntimePlugin = require('puppeteer-extra-plugin-stealth/evasions/chrome.runtime');
const ChromeCsiPlugin = require('puppeteer-extra-plugin-stealth/evasions/chrome.csi');
const ChromeLoadTimePlugin = require('puppeteer-extra-plugin-stealth/evasions/chrome.loadTimes');
const IFrameContentWindowPlugin = require('puppeteer-extra-plugin-stealth/evasions/iframe.contentWindow');
const MediaCodecsPlugin = require('puppeteer-extra-plugin-stealth/evasions/media.codecs');
const NavigatorHardwarePlugin = require('puppeteer-extra-plugin-stealth/evasions/navigator.hardwareConcurrency');
const NavigatorLanguagesPlugin = require('puppeteer-extra-plugin-stealth/evasions/navigator.languages');
const NavigatorPermissionsPlugin = require('puppeteer-extra-plugin-stealth/evasions/navigator.permissions');
const NavigatorPlugins = require('puppeteer-extra-plugin-stealth/evasions/navigator.plugins');
const WebdriverPlugin = require('puppeteer-extra-plugin-stealth/evasions/navigator.webdriver');
const UserAgentPlugin = require('puppeteer-extra-plugin-stealth/evasions/user-agent-override');
const SourceUrlPlugin = require('puppeteer-extra-plugin-stealth/evasions/sourceurl');
const WebglVendorPlugin = require('puppeteer-extra-plugin-stealth/evasions/webgl.vendor');
const WindowOuterDimensionsPlugin = require('puppeteer-extra-plugin-stealth/evasions/window.outerdimensions');
const PuppeteerUserPreferencePlpugin = require('puppeteer-extra-plugin-user-preferences');
const PuppeteerUserDataDirPlugin = require('puppeteer-extra-plugin-user-data-dir');

//puppeteer.use(StealthPlugin())

module.exports = class BotBrowserService {
  browserOptions;
  browser;
  page;
  logs;
  executablePath;

  constructor(options) {
    this.browserOptions = options;
    //this.userAgent = options.userAgent ? options.userAgent : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36';
    //this.pdfPath = './pdf'
    this.browser = null;
  }


  async openBrowser() {
    const puppeteer = addExtra(puppeteerVanilla)
    const plugins = [
      StealthPlugin(),
      DefaultArgsPlugin(),
      ChromeAppePlugin(),
      ChromeRuntimePlugin(),
      ChromeLoadTimePlugin(),
      NavigatorHardwarePlugin(),
      ChromeCsiPlugin(),
      IFrameContentWindowPlugin(),
      MediaCodecsPlugin(),
      NavigatorLanguagesPlugin(),
      NavigatorPermissionsPlugin(),
      NavigatorPlugins(),
      SourceUrlPlugin(),
      WebdriverPlugin(),
      UserAgentPlugin(),
      WebglVendorPlugin(),
      WindowOuterDimensionsPlugin(),
      PuppeteerUserPreferencePlpugin(),
      PuppeteerUserDataDirPlugin()
    ];

    try {
      console.log(StealthPlugin().availableEvasions)

      for (const plugin of plugins) {
        puppeteer.use(plugin);
      }

      this.browser = await puppeteer.launch({
        executablePath: this.executablePath,
        //executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', // for mac os User
        //executablePath: 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
        headless: this.browserOptions.headless ? this.browserOptions.headless : false,
        ignoreDefaultArgs: ['--enable-automation'],
        userDataDir: __dirname + `/userData/${this.browserOptions.account._id}`,
        args: [
          //'--disable-gpu',
          '--window-position=0,0',
          '--disable-infobars',
          "--no-sandbox",
          '--disable-setuid-sandbox',
          '--disable-site-isolation-trials',
        ],
        ignoreHTTPSErrors: true,
      })

      const [page] = await this.browser.pages()

      await page.setBypassCSP(true)
      page.setDefaultNavigationTimeout(0)
      page.setDefaultTimeout(0)
      await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36 Edg/100.0.1185.29')
      this.page = page;
      return Promise.resolve({ browser: this.browser, page })
    } catch (err) {
      console.log(err);
      //this.browser.close();
      return Promise.reject(err)
    }

  }

}

Compile: bytenode -c main.js browser.js googleLogin.js and run: node index.js.

So, clearly the issue is not in bytenode itself. It might be in bytenode-webpack-plugin or webpack in general.

Answer 5 · 2022-04-10T11:44:18.000Z

ah I see, thanks for so much effort debugging my code. looks like I need to find a way to solve webpack issue. @OsamaAbbas 🙇

Answer 6 · 2022-04-10T14:38:32.000Z

Please note that the Compiling Main Process Code[BytenodeWebpackPlugin] step generates many errors like this one:

⠋ Compiling Main Process CodeError:  evalmachine.<anonymous>:1
(function (exports, require, module, __filename, __dirname) { #!/usr/bin/env node
                                                              ^

SyntaxError: Invalid or unexpected token
    at new Script (node:vm:100:7)
    at Object.compileCode (/home/osama/Desktop/Dev/bytenode-electron-puppeteer-error/node_modules/bytenode/lib/index.js:28:18)
    at Socket.<anonymous> (/home/osama/Desktop/Dev/bytenode-electron-puppeteer-error/node_modules/bytenode/lib/cli.js:101:41)
    at Socket.emit (node:events:406:35)
    at endReadableNT (node:internal/streams/readable:1343:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

I could not figure out why or where they occur, but it's clear that BytenodeWebpackPlugin is trying to "bundle" some .js files that start with a shebang symbol. This could be the issue here.

Answer 7 · 2022-04-10T14:58:33.000Z

These errors happen when compiling main/native_modules/cli.jsc and main/native_modules/cli1.jsc files. Again, I could not easily figure out what are they. I modified bytenode to remove the shebang symbol in every possible relevant location, but this error still occurs.

Maybe @herberttn can figure this out. I have next to zero experience with webpack.

Answer 8 · 2022-04-10T19:04:37.000Z

These errors happen when compiling main/native_modules/cli.jsc and main/native_modules/cli1.jsc files. Again, I could not easily figure out what are they. I modified bytenode to remove the shebang symbol in every possible relevant location, but this error still occurs.

Maybe @herberttn can figure this out. I have next to zero experience with webpack.

after digging the internet, i found this puppeteer serializable issue on another packaging module like PKG.
https://stackoverflow.com/questions/59021700/passed-function-is-not-well-serializable

I still don't know why but some puppeteer command work but some not working as expected (return : Error: Passed function is not well-serializable!) for example:

page.waitForSelector('#identifierId' => Not Working
page.type('#identifierId', => Not working

page.keyboard.press('Enter') => WORKING

so I had to remove page.waitForSelector then update the form input with page.evaluate(

before:
await this.browser.page.type('#identifierId', this.browser.browserOptions.account.email, { delay: 0 })
after:

await eval(`this.browser.page.evaluate(
      () => { (document.querySelector('#identifierId').value = '${this.browser.browserOptions.account.email}'); }
)`);

and now it's working without any error

I still don't know it's related to the puppeteer itself or the webpack plugin. because I don't really know how both module works in depth :))