invalid session id when running as scraper on Heroku
albertoramires opened this issue · 1 comments
Issue
invalid session id when running as scraper on Heroku, running fine locally, visiting https://google.com or JSON endpoints runs just fine, which makes me wonder if it's a JS issue killing the session, but why would it do that even with js_errors: false
? 🤔
Maybe related to: #473
Details
Elixir: Elixir 1.12.3
Erlang: Erlang/OTP 24
OS: Heroku
Webdriver (selenium, chromedriver, geckodriver, etc): chromedriver
Browser (Chrome, Firefox, Safari, etc): Chrome
Test Code & HTML
Trying to run Wallaby as a scraper on Heroku but getting the following error when running a simple visit:
{:wallaby, "~> 0.29.0"}
Same thing with default config and this one:
config :wallaby,
chromedriver: [
capabilities: %{
javascriptEnabled: true,
loadImages: false,
version: "",
rotatable: false,
takesScreenshot: true,
cssSelectorsEnabled: true,
nativeEvents: false,
platform: "ANY",
unhandledPromptBehavior: "accept",
loggingPrefs: %{
browser: "ALL"
},
chromeOptions: %{
args: [
"--no-sandbox",
"window-size=1280,800",
"--disable-gpu",
"--headless",
"--fullscreen",
"--user-agent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
]
}
}
],
js_errors: false
{:ok, session} = Wallaby.start_session()
session |> visit(url)
2022-04-04T18:16:41.793357+00:00 app[web.1]: ** (RuntimeError) invalid session id
2022-04-04T18:16:41.793369+00:00 app[web.1]: (wallaby 0.29.1) lib/wallaby/httpclient.ex:136: Wallaby.HTTPClient.check_for_response_errors/1
2022-04-04T18:16:41.793370+00:00 app[web.1]: (wallaby 0.29.1) lib/wallaby/httpclient.ex:56: Wallaby.HTTPClient.make_request/5
2022-04-04T18:16:41.793370+00:00 app[web.1]: (wallaby 0.29.1) lib/wallaby/webdriver_client.ex:535: Wallaby.WebdriverClient.log/1
2022-04-04T18:16:41.793370+00:00 app[web.1]: (wallaby 0.29.1) lib/wallaby/driver/log_checker.ex:8: Wallaby.Driver.LogChecker.check_logs!/2
2022-04-04T18:16:41.793371+00:00 app[web.1]: (wallaby 0.29.1) lib/wallaby/browser.ex:1232: Wallaby.Browser.visit/2
I have both https://github.com/heroku/heroku-buildpack-google-chrome and https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-chromedriver in my buildpacks.
After much debugging, adding --disable-dev-shm-usage
to the chromeOptions args solves the issue due to: https://devcenter.heroku.com/changelog-items/1085