mozilla/geckodriver

Support for Selenium’s logging interface

graingert opened this issue · 32 comments

graingert commented

The logging interface supported by Selenium FirefoxDriver isn’t available in Marionette.

I'd like support for the client.log('browser') call.

jgraham commented

This isn't in the W3C spec at this time, so we are delaying support until the behaviour is well specified. But your request is noted.

graingert commented

Ah I see, is there any way to follow the development of the spec (mailing
list github etc)?

On 18 Oct 2016 10:57, "jgraham" notifications@github.com wrote:

This isn't in the W3C spec at this time, so we are delaying support until
the behaviour is well specified. But your request is noted.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#284 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZQTCR96Pk73MW0q3LzDrFlKXKIz25kks5q1JgEgaJpZM4KZliw
.

jgraham commented

On 18/10/16 11:24, Thomas Grainger wrote:

Ah I see, is there any way to follow the development of the spec (mailing
list github etc)?

github.com/w3c/webdriver or the public-browser-tools-testing@w3.org
mailing list.

graingert commented

Here's the specific issue: w3c/webdriver#406

Deleted user commented

I'm looking forward to this feature too.

shepmaster commented

I don't suppose that there is some temporary alternative we can use? Verifying that there are no JS errors in our automated tests is a very nice feature.

  • 👍10
lucaspiller commented

@shepmaster I've been looking at this today, as we also want to verify that no JS errors are thrown.

My solution probably won't go down too favourably in this repo, but it does the trick: Switch to using Chrome for integration tests, as chromedriver supports that standard Selenium browser logging interface.

  • 👍3
andreastt commented

@lucaspiller Chrome uses a standards incompatible version of the logging interface and we have no intention of implementing that.

A brief update on this is that we discussed logging at the last WebDriver working group meeting and I wrote down some quick words here: https://sny.no/2017/12/wdupdate

whimboo commented

Given that someone filed a bug for that, we now have https://bugzilla.mozilla.org/show_bug.cgi?id=1453962.

  • 👍4
yonjah commented

@andreastt any news on this one ?
If I understand it correctly the recommendation currently by w3c team is to use the chromedriver w3c/webdriver#710 (comment) . It doesn't look like this has any priority to be entered into the standard w3c/webdriver#406

I was really hopping to have firefox as my main target for automated testing but not having logs means its not going to be practical

Herst commented

@yonjah If I understand it correctly the WebDriver clients themselves use more powerful protocols to communicate with the applications. In the case of ChromeDriver it's CDP and in the case of Firefox it's Marionette, so while a certain feature might not be part of WebDriver it might still be part of the browser-specific control protocol and you can make use it of it in this way.

andreastt commented

@yonjah In fear of derailing this discussion, the WebDriver WG did talk about logging at the previous F2F in Lyon.

The fear the WG has is that the logging API as implemented in Chrome is limited by the pull-push model restriction of HTTP: it builds up a log cache in a ring buffer internally, and then returns the cache when requested to by the client. This runs the risk of missing out on log entries (because of the ring buffer) and is generally speaking a feature not suited to a blocking pull-push transport mechanism such as HTTP/1.1.

Since this issue was filed there has been some interest amongst vendors to allow CDP as an upgrade mechanism in WebDriver, so a large part of the discussion in Lyon centred around what a bidirectional asynchronous protocol might enable in terms of features. An obvious publish-subscribe candidate would be logging.

At the moment I see little hope for geckodriver implementing the Selenium logging API if (a) it is not part of the WebDriver standard, and (b) a log service on top of CDP exists and there is interest in the WG for an upgrade mechanism to that.

  • 👍2
yonjah commented

@Herst Could you please explain how can I obtain the network log and console error log when I'm running my tests with selenium and firefox ?

I have a few flaky tests that fail with blank screen (from a screenshot) though page html seem to load. I can only guess there some network or JS error is preventing the app from loading but I can't debug it with no access to the logs. I can switch our initial tests to run with chrome but I was hoping firefox will be the main target for our testing.

@andreastt as long as there is some way to get this logs I guess from a user perspective it doesn't matter that much. But I think it would be useful to make it clear how to get this logs ( what is CDP? how do I use the log service to get this logs ?) on since it seem like the really easy solution is just to switch to chrome, which doesn't help Mozilla or the standard.

Herst commented

@yonjah I am sorry don't know how to do it in Firefox, I just outlined how it might be theoretically possible. I know that it is possible in Chrome through CDP (Chrome Devtools Protocol), a more powerful but also more low-level protocol for programmable interaction with Chrom(e|ium). Clients like Selenium for Python then allow you to execute these commands right away. I have no idea whether the same is possible with Mozilla's equivalent for Firefox, I would also be interested to know so Firefox will be less of a second-grade browser in our CI setup.

BTW, I personally would always test in both browsers. (And maybe even Safari if you want an extra challenge.)

@andreastt I understand the reasoning, and yet "perfect is the enemy of good"… From a perspective of a testing engineer writing automated tests for large-scale web projects is simply better in Chrome because thanks to this feature I can catch potential problems which show up in the error console but not in the tests (e.g. Javascript errors which lead to graphical/interaction glitches but don't affect any of the tested functionality, or missing resources…, and yes I know that former I could catch using pixel testing and latter by looking at traffic e.g. with the help of a proxy, but that would mean a lot of effort for unknown gain).

yonjah commented

Thanks @Herst
It seem like in webdriverjs I can use sendDevToolsCommand to send CDP commands to chrome but I can't see any similar API commands available when using the firefox driver so I assume it is not supported.

I plan to test on a few more browsers once I get all the kinks out of our UI testing in general but it doesn't make much sense to do it now. Also I only need this feature for debugging the tests at the moment, but it makes sense to fail the tests if a the logs contain unexpected errors. If I'll decide to add such a feature to our tests it means they will not be able to run with firefox which is a shame.

I guess I'll go back and try to reproduce the flakiness with chrome for now

snowymike commented

we have to rotate browsers in our testing anyway (ie, firefox, chrome are the 3 we use), and for my team we've just accepted that chrome is the only one we can get meaningful console logs from. fortunately we haven't had any JS errors in our product (yet) that were firefox specific and that couldn't also be figured out from the screenshot and HTML we also save off (in all 3 browsers) when a test fails. but it'd be great if we could get firefox console logs like we can for chrome, someday.

whimboo commented

Please note that with the upcoming Firefox 65 release there will be the new preference devtools.console.stdout.content users can set which will route every Console API call (like info(), warn(), and error()) to stdout. That means the output will end-up in the geckodriver log. It's not perfect but at least a start to get some of the logs.

  • 👍20
graingert commented
whimboo commented

If those aren't passed trough the Console API then no.

LeHack commented

Ok, so here's a solution for Firefox that I designed and my company decided to give away as open source: https://github.com/hurracom/WebConsoleTap/releases
For now it only takes care of console.log/error/info/warn, but it's quite trivial to extend it to catch other things as well.

Hope it helps some of you.

  • 👍6
rsingh-03 commented

Please note that with the upcoming Firefox 65 release there will be the new preference devtools.console.stdout.content users can set which will route every Console API call (like info(), warn(), and error()) to stdout. That means the output will end-up in the geckodriver log. It's not perfect but at least a start to get some of the logs.

Is there any way I can specify gecko_log_path? by default ,log file is getting created in working dir.But I want to change it's location using set_preferences and not while instantiating webdriver.firefox()?

graingert commented
testphreak commented

Please note that with the upcoming Firefox 65 release there will be the new preference devtools.console.stdout.content users can set which will route every Console API call (like info(), warn(), and error()) to stdout. That means the output will end-up in the geckodriver log. It's not perfect but at least a start to get some of the logs.

Hi @whimboo, thank you to the team for this feature! It isn't clear to me if it is possible to capture Firefox console logs from RemoteWebDriver (running in a Selenium Grid) instead of locally. If it is possible, could you share how. Also, is it possible to only log console errors to stdout?

whimboo commented

All those logs go to the default log which should be the same geckodriver uses. But maybe there are differences in the Selenium bindings. Just try it out. I cannot help here given that this is more a question about Selenium.

mandric commented

For JS folks wrestling with this, here is the minimum config I came up using selenium-webdriver to get console.log output from a web page.

The trick was FF >= 65 and the preference from above devtools.console.stdout.content plus setStdio call.

const { Builder, logging } = require('selenium-webdriver')
const firefox = require('selenium-webdriver/firefox');
const chrome = require('selenium-webdriver/chrome');

// provide access to console logs in Chrome
const prefs = new logging.Preferences()
prefs.setLevel(logging.Type.BROWSER, logging.Level.ALL);

const driver = module.exports = new Builder()
  .forBrowser('firefox')
  .setLoggingPrefs(prefs)
  .setFirefoxOptions(
    new firefox.Options()
      .headless()
      // provide access to console logs in Firefox
      .setPreference('devtools.console.stdout.content', true)
  )
  .setChromeOptions(new chrome.Options().headless())
  .setChromeService(
    new chrome.ServiceBuilder()
      //.enableVerboseLogging()
      .setStdio('inherit')
  )
  .setFirefoxService(
    new firefox.ServiceBuilder()
      //.enableVerboseLogging()
      .setStdio('inherit')
  )
  .build()

if (require.main === module) {
  // run test if not called via require
  (async function() {
    try {
      await driver.executeScript(`
        console.log('AHOY WEB CONSOLE!')
      `);
      try {
        // Needed for Chrome. Firefox throws here, will not implement.
        // https://github.com/mozilla/geckodriver/issues/284
        await driver.manage().logs().get(logging.Type.BROWSER).then(console.log);
      } catch(err) {
        console.warn(err)
      }
    } catch(err) {
      console.error(err)
    } finally {
      if (driver) {
        await driver.quit()
      }
    }
  })();
}
  • 👍10
  • 🚀3
hathiphant commented

Just for simplicity, isn't it possible to do something simple on HTTP with Server-Sent Events endpoints, without adding the complexity of a BiDi protocol on WebSocket.

When you open a window with WebDriver, you configure optionally log type/level and you obtain in response the URL on geckodriver of an SSE endpoint with events of navigation. This is the only SSE endpoint living as long as window and keeping only history of navigation (like a browser). In the events of navigation, the message contains the URL of each new location in the window and also the URL of an SSE endpoint sending the log events occurring for this page but without warranty to be available more than the time where the page is currently loaded in the window (this does not need more cache of events than devtools currently use when launched on a page).

POST http://127.0.0.1:4444/session
{"capabilities", {"alwaysMatch": {"browserName", "firefox", "logs": {"javascript": "error", "css": "warning", "request": "trace"}}}}
responding with:
{"value": {"sessionId": "$sessionId", "navigation": "$navigation_endpoint"}}

GET http://127.0.0.1:4444/navigation/$navigation_endpoint
This SSE endpoint is only available when window is open and sends SSE events like:

event: location
data: {"url": "...", "logs": "$pagelog_endpoint"}

GET http://127.0.0.1:4444/logs/$pagelog_endpoint
This SSE endpoint is only available when the driven window is on the corresponding page; after that, geckodriver is free to forget logs and to respond HTTP 410 Gone or 404. It sends SSE events of logs, like:

event: request
data: {"level": "trace", "url": "..."}

event: css
data: {"level": "error", "url": "...", "line": ..., "column": ..., "message": "..."}

Just my opinion,
Thanks for reading.

whimboo commented

Note that with Selenium 4 there will be support for CDP (Chrome DevTools Protocol). I'm currently working on making geckodriver to offer an opt-in for a websocket URL that can be used with our own implementation of a subset of the CDP APIs (called Remote Protocol. We already have a good amount of APIs (commands and events) implemented, and we want to see that especially the logging events will work. That means it can be used until the WebDriver Bidi support is available by sometime next year.

PhilippSelenium commented

@whimboo Should it already work with selenium 4.0 and geckodriver 30.0?

  • 👍2
whimboo commented

Selenium 4.0.0 makes use of the CDP implementation for both Firefox and Chrome. Note that this a temporary solution to allow users to subscribe to log events. Currently we are working on WebDriver BiDi support for log.entryAdded which should be fully available by at latest end of this year.

The Selenium implementation has been built to cause a minimal impact on consumers when the move from CDP to BiDi will happen. So yes, you should be able to get started but note that our CDP implementation for log events is not complete, and won't ever be compared to the WebDriver BiDi standard implementation.

Given that geckodriver will not be used for WebDriver BiDi but Firefox itself will contain the full implementation I do not see a need to further track this issue. As such I'm going to close it now.

I hope you all will enjoy the soon upcoming feature!

PhilippSelenium commented

@whimboo Thanks, but this answer caused more confusion for me than it claryfied:

So CDP will not be (completely) supported but the BiDi will not be supported by geckodriver since it will be an inherent feature of firefox??

What has to happen on the selenium side (or any other side) to get the console log via client.log("browser")?

whimboo commented

Right, geckodriver is only used for WebDriver HTTP, and as helper to opt-in to WebDriver BiDi via the webSocketUrl capability. Everything else will be directly implemented in Firefox itself, and communication will happen via its own WebSocket connection.

I cannot speak about the Selenium side given that this is its own project. But what you want is:

  1. Start a WebDriver session via geckodriver with webSocketURL set to true.
  2. Retrieve the server address for WebDriver BiDi via the webSocketURL capability.
  3. Use a WebSocket client to connect to that address.
  4. Subscribe for log.entryAdded events.
  5. Handle log.entryAdded events in your code

Note that there is still the hen and egg problem with not having a proper BiDi client available yet. But AFAIK Selenium folks want to integrate the BiDi feature probably in the 4.1 release.

To give you an impression here is some code from the web-platform-tests repository with a basic test for log.entryAdded that checks console messages.

  • 👍1