ulixee/secret-agent

failure to substitute :method (with GET?) sometimes in MitmRequestAgent

andynuss opened this issue · 1 comments

I don't know if this has anything to do with this issue (ulixee/hero#161), but
on one of the random urls that I was testing with happened to be a PDF, and my default browser plugin
seemingly failed to substitute the :method template param with GET, before issuing the GET, I assume
to download the PDF. I only saw it happen once so far but will keep a watch on it. I'm betting it had something to do with my low, horse-powered ec2 instance
that I am testing on.

TypeError [ERR_INVALID_HTTP_TOKEN]: Header name must be a valid HTTP token [":method"]
    at ClientRequest.setHeader (_http_outgoing.js:563:3)
    at new ClientRequest (_http_client.js:262:14)
    at Object.request (https.js:370:10)
    at MitmRequestAgent.http1Request (/home/ec2-user/app/node_modules/mitm/lib/MitmRequestAgent.ts:217:32)
    at MitmRequestAgent.request (/home/ec2-user/app/node_modules/mitm/lib/MitmRequestAgent.ts:92:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at HttpRequestHandler.createProxyToServerRequest (/home/ec2-user/app/node_modules/mitm/handlers/BaseHttpHandler.ts:61:23)
    at HttpRequestHandler.onRequest (/home/ec2-user/app/node_modules/mitm/handlers/HttpRequestHandler.ts:37:36)
    at Function.onRequest (/home/ec2-user/app/node_modules/mitm/handlers/HttpRequestHandler.ts:276:5)
------REMOTE CORE---------------------------------
    at Function.reviver (/home/ec2-user/app/node_modules/commons/TypeSerializer.ts:208:26)
    at JSON.parse (<anonymous>)
    at Function.parse (/home/ec2-user/app/node_modules/commons/TypeSerializer.ts:24:17)
    at WebSocket.<anonymous> (/home/ec2-user/app/node_modules/client/connections/RemoteConnectionToCore.ts:67:42)
    at WebSocket.emit (events.js:400:28)
    at WebSocket.emit (domain.js:475:12)
    at Receiver.receiverOnMessage (/home/ec2-user/app/node_modules/ws/lib/websocket.js:983:20)
    at Receiver.emit (events.js:400:28)
    at Receiver.emit (domain.js:475:12)
    at Receiver.dataMessage (/home/ec2-user/app/node_modules/ws/lib/receiver.js:517:14)
------CONNECTION----------------------------------
    at new Resolvable (/home/ec2-user/app/node_modules/commons/Resolvable.ts:17:18)
    at Object.createPromise (/home/ec2-user/app/node_modules/commons/utils.ts:68:10)
    at RemoteConnectionToCore.createPendingResult (/home/ec2-user/app/node_modules/client/connections/ConnectionToCore.ts:328:31)
    at RemoteConnectionToCore.internalSendRequestAndWait (/home/ec2-user/app/node_modules/client/connections/ConnectionToCore.ts:253:43)
    at RemoteConnectionToCore.sendRequest (/home/ec2-user/app/node_modules/client/connections/ConnectionToCore.ts:156:17)
    at Object.cb (/home/ec2-user/app/node_modules/client/lib/CoreCommandQueue.ts:104:26)
    at Queue.next (/home/ec2-user/app/node_modules/commons/Queue.ts:82:19)
------CORE COMMANDS-------------------------------
    at Queue.run (/home/ec2-user/app/node_modules/commons/Queue.ts:35:19)
    at CoreCommandQueue.run (/home/ec2-user/app/node_modules/client/lib/CoreCommandQueue.ts:100:8)
    at CoreTab.goto (/home/ec2-user/app/node_modules/client/lib/CoreTab.ts:92:36)
    at Tab.goto (/home/ec2-user/app/node_modules/client/lib/Tab.ts:160:36)
    at /home/ec2-user/app/src/scraper.js:245:32

--------------------------------------------------
------default-session-67--------------------------
------e2630310-39bb-11ec-88da-731dedebd20f--------
--------------------------------------------------

I was able to reproduce it by chance a single time in standalone testing.

The url happened to be this:

https://www.mragheb.com/Restoring%20The%20Global%20Equatorial%20Ocean%20Current%20Using%20Nuclear%20Excavation.pdf

Are you running a custom browser plugin here? Looks like an http2 header is being sent to an http1 request. Can you send that session database? e2630310-39bb-11ec-88da-731dedebd20f.db