cdp_webdriver: A Java repository from sergueik

Info

This project contains test scenarios practicing Java accessing Chrome Devtools API during Selenium test without upgrading the Selenium driver to alpha release 4.0.x

The code was developed from replica of ahajamit/chrome-devtools-webdriver-integration Chrome DevTools WebDriver integration project with borrowing more utils and test scenarios (the upstream project development stopped in Dec 2019).

For accessing the Chrome Devtools API after ugrading the Selenium driver to alpha release 4.0.x see Selenium CDP project

Operation

The custom driver extension examines the chrome log file located in

System.getProperty("user.dir")  + "/target/chromedriver.log"

and finds the line

[1587217990.273][INFO]: Launching chrome: "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-extensions --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --ignore-certificate-errors --ignore-ssl-errors=true --log-level=0 --no-first-run --password-store=basic --remote-debugging-port=0 --ssl-protocol=any --start-maximized --test-type=webdriver --use-mock-keychain --user-data-dir="C:\Users\Serguei\AppData\Local\Temp\scoped_dir5740_1744005879" data:,
[1587217990.738][DEBUG]: DevTools HTTP Request: http://localhost:51086/json/version

then probes the chrome browser available sockets by sending

http://localhost:51086/json

to the port the browser DevTools is listening to, in this case 51086. extracting the webSocketDebuggerUrl from the response

[
  {
    "description": "",
    "devtoolsFrontendUrl": "/devtools/inspector.html?ws=localhost:10000/devtools/page/31BCAA3827B696A389EFE222BE7F9B0E",
    "id": "31BCAA3827B696A389EFE222BE7F9B0E",
    "title": "Service Worker https://www.google.com.sg/maps/preview/sw?hl=en",
    "type": "service_worker",
    "url": "https://www.google.com.sg/maps/preview/sw?hl=en",
    "webSocketDebuggerUrl": "ws://localhost:10000/devtools/page/31BCAA3827B696A389EFE222BE7F9B0E"
  }
]

and constructs a socket for that port using com.neovisionaries.ws.client.WebSocket. The actual CDP commands and responses are posted to and read from that socket. The MessageBuilder class is used to deal with JSON conversion of session id and various message parameters using gson, e.g.

private static String buildMessage(int id, String method Map<String, Object> params) {
  final Gson gson = new Gson();
  message = new Message(id, method);
  for (String key : params.keySet()) {
    message.addParam(key, params.get(key));
  }
  return gson.toJson(message);
}

where Message is a generic class with properties id, method, and params. this for an e.g. Emulation.setGeolocationOverride creates payload which looks like:

{
  "id": 196822,
  "method": "Emulation.setGeolocationOverride",
  "params": {
    "latitude": 37.42229,
    "longitude": -122.084057,
    "accuracy": 100
  }
}

Supported CDP API

Every CDP API becomes a static "message builder" build<CDP API>Message method of example.messaging.MessageBuilder class with the following methods defiened:

buildActivateTargetMessage
buildAttachToTargetMessage
buildBasicHttpAuthenticationMessage
buildBrowserVersionMessage
buildClearBrowserCacheMessage
buildClearBrowserCookiesMessage
buildClearDataForOriginMessage
buildCloseTargetMessage
buildCreateTargetMessage
buildCustomRuntimeEvaluateMessage
buildDeleteCookiesMessage
buildDescribeNodeMessage
buildDetachFromTargetMessage
buildDOMEnableMessage
buildEmulationResetPageScaleMessage
buildEmulationSetDeviceMetricsMessage
buildEmulationSetVisibleSizeMessage
buildEnableLogMessage
buildEnableRuntimeMessage
buildGeoLocationMessage
buildGetAllCookiesMessage
buildGetBrowserContextMessage
buildGetContinueInterceptedRequestEncodedMessage
buildGetContinueInterceptedRequestMessage
buildGetDocumentMessage
buildGetOuterHTMLMessage
buildGetResponseBodyForInterceptionMessage
buildGetResponseBodyMessage
buildGetTargetsMessage
buildNetWorkEnableMessage
buildNetWorkSetExtraHTTPHeadersMessage
buildObserveBackgroundServiceMessage
buildOverlayEnableMessage
buildOverlayHighlightFrameMessage
buildPageAddScriptToEvaluateOnNewDocumentMessage
buildPageGetFrameOwnerMessage
buildPageGetFrameTreeMessage
buildPageRemoveScriptToEvaluateOnNewDocument
buildPerformanceDisableMessage
buildPerformanceEnableMessage
buildPerformanceGetMetricsMessage
buildPrintPDFMessage
buildQuerySelectorMessage
buildRequestInterceptorEnabledMessage
buildRequestInterceptorPatternMessage
buildRuntimeEvaluateMessage
buildSendObservingPushMessage
buildSendPushNotificationMessage
buildServiceWorkerEnableMessage
buildServiceWorkerInspectMessage
buildSetTimeDomainMessage
buildSetUserAgentOverrideMessage
buildTakeElementScreenShotMessage
buildTakePageScreenShotMessage
buildTargetInfoMessage
buildTimezoneOverrideMessage

The tests have been provided for practically every method from the above

Docker Testing

Currently the test described in the original repository author's blog does not appear to work:

Setup

download stock Docker image with chrome and selenium proxy

docker pull selenium/standalone-chrome

Note: the defauld docker image is based on ubuntu and its size is 908 Mb compared to some 384 Mb of custom alpine based image with chromium browser.

run the Docker container with additional port published

docker run -d --expose=9222 -p 4444:4444 -p 0.0.0.0:9222:9222 --name selenium-standalone-chrome -v /dev/shm:/dev/shm selenium/standalone-chrome

alternatively can specify custom debugging port and omit volume:

docker container prune -f
docker run -d --expose=10000 -p 4444:4444 -p 0.0.0.0:10000:10000 --name selenium-standalone-chrome selenium/standalone-chrome

Run Test

run the Docker tests

mvn test -DdebugPort=10000

will see the error:

???

Alternatively, have JDK and maven in the Docker container and run the tests completely in the container from mapped volume (this is how it is done in maven/jdk8 apline base image).

TODO

in headless run:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.923 sec

Results :

Failed tests:   test1(example.BrowserDownloadTest): (..)
  test2(example.BrowserDownloadTest): (..)
  test1(example.BasicAuthHeadersFailingTest): (..)
  test1(example.BasicAuthHeadersTest): (..)
  test1(example.PageDownloadTest): (..)
  test(example.ShadowRootTest): (..)

Tests in error:
  test1(example.PerformanceMetricsTest): JSONArray[0] not found.
  test2(example.GeolocationOverrideTest): Expected condition failed: waiting for example.utils.UIUtils$$Lambda$221/844872102@64ec1459 (tried for 120 second(s) with 1000 milliseconds interval)
  test3(example.GeolocationOverrideTest): Expected condition failed: waiting for example.utils.UIUtils$$Lambda$221/844872102@242c4a94 (tried for 120 second(s) with 1000 milliseconds interval)
  test1(example.NetworkTrackingTest): java.lang.RuntimeException: No message received with id: 945244
  test1(example.DeviceMetricsOverrideTest): no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="content-base"]//table//th[contains(text(),"VIEWPORT-WIDTH")]/../td"}(..)
  test2(example.DeviceMetricsOverrideTest): no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="content-base"]//table//th[contains(text(),"VIEWPORT-WIDTH")]/../td"}(..)
  test2(example.IndirectGeolocationOverrideTest): Expected condition failed: waiting for example.utils.UIUtils$$Lambda$221/844872102@364f0a6f (tried for 120 second(s) with 1000 milliseconds interval)

Tests run: 71, Failures: 6, Errors: 7, Skipped: 12

in foreground test:

Failed tests:   test4(example.PrintPDFTest): (..)
  test1(example.DeviceMetricsOverrideTest): (..)
  test2(example.DeviceMetricsOverrideTest): (..)
  test1(example.BasicAuthHeadersTest): (..)

Tests in error: 
  test2(example.GeolocationOverrideTest): Expected condition failed: waiting for example.utils.UIUtils$$Lambda$238/700286427@509e4902 (tried for 120 second(s) with 1000 milliseconds interval)
  test3(example.GeolocationOverrideTest): Expected condition failed: waiting for example.utils.UIUtils$$Lambda$238/700286427@4d8458a1 (tried for 120 second(s) with 1000 milliseconds interval)
  test2(example.IndirectGeolocationOverrideTest): Expected condition failed: waiting for example.utils.UIUtils$$Lambda$238/700286427@564519de (tried for 120 second(s) with 1000 milliseconds interval)

Tests run: 70, Failures: 4, Errors: 3, Skipped: 12

Author

Serguei Kouzmine

sergueik/cdp_webdriver