elixir-wallaby/wallaby

invalid session id when running as scraper on Heroku

albertoramires opened this issue · 1 comments

Issue

invalid session id when running as scraper on Heroku, running fine locally, visiting https://google.com or JSON endpoints runs just fine, which makes me wonder if it's a JS issue killing the session, but why would it do that even with js_errors: false? 🤔
Maybe related to: #473

Details

Elixir: Elixir 1.12.3
Erlang: Erlang/OTP 24
OS: Heroku
Webdriver (selenium, chromedriver, geckodriver, etc): chromedriver
Browser (Chrome, Firefox, Safari, etc): Chrome

Test Code & HTML

Trying to run Wallaby as a scraper on Heroku but getting the following error when running a simple visit:

{:wallaby, "~> 0.29.0"}

Same thing with default config and this one:

config :wallaby,
  chromedriver: [
    capabilities: %{
      javascriptEnabled: true,
      loadImages: false,
      version: "",
      rotatable: false,
      takesScreenshot: true,
      cssSelectorsEnabled: true,
      nativeEvents: false,
      platform: "ANY",
      unhandledPromptBehavior: "accept",
      loggingPrefs: %{
        browser: "ALL"
      },
      chromeOptions: %{
        args: [
          "--no-sandbox",
          "window-size=1280,800",
          "--disable-gpu",
          "--headless",
          "--fullscreen",
          "--user-agent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
        ]
      }
    }
  ],
  js_errors: false
{:ok, session} = Wallaby.start_session()

session |> visit(url)
2022-04-04T18:16:41.793357+00:00 app[web.1]: ** (RuntimeError) invalid session id

2022-04-04T18:16:41.793369+00:00 app[web.1]:     (wallaby 0.29.1) lib/wallaby/httpclient.ex:136: Wallaby.HTTPClient.check_for_response_errors/1

2022-04-04T18:16:41.793370+00:00 app[web.1]:     (wallaby 0.29.1) lib/wallaby/httpclient.ex:56: Wallaby.HTTPClient.make_request/5

2022-04-04T18:16:41.793370+00:00 app[web.1]:     (wallaby 0.29.1) lib/wallaby/webdriver_client.ex:535: Wallaby.WebdriverClient.log/1

2022-04-04T18:16:41.793370+00:00 app[web.1]:     (wallaby 0.29.1) lib/wallaby/driver/log_checker.ex:8: Wallaby.Driver.LogChecker.check_logs!/2

2022-04-04T18:16:41.793371+00:00 app[web.1]:     (wallaby 0.29.1) lib/wallaby/browser.ex:1232: Wallaby.Browser.visit/2

I have both https://github.com/heroku/heroku-buildpack-google-chrome and https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-chromedriver in my buildpacks.

After much debugging, adding --disable-dev-shm-usage to the chromeOptions args solves the issue due to: https://devcenter.heroku.com/changelog-items/1085