ulixee/secret-agent

Clicking on a canvas inside an iframe

Leandro-Amorim opened this issue · 6 comments

Leandro-Amorim commented

I am clicking on a canvas element within an iframe on a page, but the program is returning an error. Below is the code I am using and the error.

	static async clickOnCanvas(agent, fX, fY) {
		const iframe = agent.document.querySelector('#bxr-iframe');
		const rect = await iframe.getBoundingClientRect();
		const x = rect.x + fX * rect.width;
		const y = rect.y + fY * rect.height;
		await agent.interact({ click: [x, y] });
	}

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^
Error [InjectedScriptError]: Error: unknown JsPathStep
    at JsPath.runJsPath (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\core\lib\JsPath.ts:112:13)
    at Interactor.lookupBoundingRect (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\core\lib\Interactor.ts:137:24)
    at DefaultHumanEmulator.moveMouseAndClick (E:\Home\Desktop\f\Web Scraping\VPSolver\plugins\default-human-emulator\index.ts:147:20)
    at DefaultHumanEmulator.playInteractions (E:\Home\Desktop\f\Web Scraping\VPSolver\plugins\default-human-emulator\index.ts:75:11)
    at CorePlugins.playInteractions (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\core\lib\CorePlugins.ts:198:7)
------REMOTE CORE---------------------------------
    at Function.reviver (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\commons\TypeSerializer.ts:208:26)
    at JSON.parse (<anonymous>)
    at Function.parse (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\commons\TypeSerializer.ts:24:17)
    at WebSocket.<anonymous> (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\connections\RemoteConnectionToCore.ts:67:42)
    at WebSocket.emit (node:events:527:28)
    at Receiver.receiverOnMessage (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\ws\lib\websocket.js:1068:20)
    at Receiver.emit (node:events:527:28)
    at Receiver.dataMessage (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\ws\lib\receiver.js:517:14)
    at E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\ws\lib\receiver.js:468:23
    at E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\ws\lib\permessage-deflate.js:308:9
------CONNECTION----------------------------------
    at new Resolvable (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\commons\Resolvable.ts:16:18)
    at Object.createPromise (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\commons\utils.ts:68:10)
    at RemoteConnectionToCore.createPendingResult (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\connections\ConnectionToCore.ts:357:31)
    at RemoteConnectionToCore.internalSendRequestAndWait (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\connections\ConnectionToCore.ts:263:43)
    at RemoteConnectionToCore.sendRequest (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\connections\ConnectionToCore.ts:164:17)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at CoreCommandQueue.sendRequest (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\lib\CoreCommandQueue.ts:150:22)
    at Object.cb (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\lib\CoreCommandQueue.ts:117:16)
    at Queue.next (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\commons\Queue.ts:82:19)
------CORE COMMANDS-------------------------------
    at Queue.run (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\commons\Queue.ts:35:19)
    at CoreCommandQueue.run (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\lib\CoreCommandQueue.ts:114:8)
    at CoreFrameEnvironment.interact (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\lib\CoreFrameEnvironment.ts:79:29)
    at Function.run (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\lib\Interactor.ts:47:21)
    at Agent.interact (E:\Home\Desktop\f\Web Scraping\VPSolver\node_modules\client\lib\Agent.ts:299:22)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Function.clickOnCanvas (E:\Home\Desktop\f\Web Scraping\VPSolver\browser.js:45:3)
    at Function.setup (E:\Home\Desktop\f\Web Scraping\VPSolver\browser.js:21:3)

--------------------------------------------------
------default-session-2---------------------------
------6df53f50-2323-11ed-9e44-ddbfe3dbe2a6--------
-------------------------------------------------- {
  pathState: { step: null, index: 0 }
}

I suspect that the error is being caused by a browser security policy. Is there any way to bypass this?

blakebyrnes commented

Hi @Leandro-Amorim, I would make sure you are generating actual "numbers" for those coordinates, and not NaN. Is there any reason you're using SecretAgent vs Hero? This project development has moved over to that repo.

Leandro-Amorim commented

Yes, the coordinates are numbers. I migrated the project to Hero and it worked. Thanks for the help! However, when I call the click function, Hero scrolls the page to the coordinates specified for the click. I know it's not related to the issue, but could you help me with this?

blakebyrnes commented

Np. Are you trying to not scroll to the element?

Leandro-Amorim commented

Exactly, I don't want the page to scroll to the clicked element.

blakebyrnes commented
Leandro-Amorim commented

Okay, thank you for the help!