/SeleniumCaptureHttpResponse

Example of how to capture an HTTP response using Selenium WebDriver, ChromeDriver, Chrome DevTools Protocol, and C#.

Primary LanguageC#

What's SeleniumCaptureHttpResponse?

SeleniumCaptureHttpResponse contains two C# .NET 8 console apps that demonstrate how to capture a HTTP responses using Selenium WebDriver, ChromeDriver, Chrome DevTools Protocol (CDP), and WebDriver BiDi. It is not a standalone application, it's a working example of how you can capture HTTP responses (e.g. JSON from API calls) on web pages, inc. 3rd party websites. Therefore, you'll need software to compile the code, e.g. Visual Studio 2022.

What's Selenium WebDriver?

Selenium WebDriver controls compatible web browsers (inc. Firefox, Chrome, Edge, Internet Explorer) natively, as a real user would. Selenium .NET API documentation can be found on their website.

What's ChromeDriver?

ChromeDriver is an executable that Selenium WebDriver uses to control Chrome.

What's Chrome DevTools Protocol?

Chrome DevTools Protocol (CDP) allows for tools to instrument, inspect, debug, and profile, Chromium, Chrome, and other Blink-based browsers. SeleniumCaptureHttpResponse uses CDP to capture HTTP responses.

What's WebDriver BiDi?

WebDriver BiDi extends WebDriver by introducing bidirectional communication. In place of the strict command/response format of WebDriver, this permits events to stream from web browsers to the controlling software, better matching the evented nature of the browser DOM. SeleniumCaptureHttpResponse uses WebDriver BiDi to capture HTTP responses.

Where could this be useful?

Testing a web page for bugs or as part of scraping a web page where capturing HTTP traffic is desired.

What's the difference between the two console apps?

Metalhead.SeleniumCaptureHttpResponse.BiDi uses WebDriver BiDi to capture HTTP responses. Metalhead.SeleniumCaptureHttpResponse.CDP uses Chrome DevTools Protocol to capture HTTP responses.

Setup instructions

  1. Install the Chrome web browser.
  2. Download the version of ChromeDriver that matches your version of Chrome. If don't need the latest version of ChromeDriver, then older versions are available. Note: 32-bit version works with 64-bit Chrome.
  3. Clone the SeleniumCaptureHttpResponse repository.
  4. Open the .NET solution in Visual Studio 2022 (or a compatible alternative).
  5. Open the appsettings.json file and edit the paths to match the location of Chrome and ChromeDriver on your PC. Note: If you set these paths to null, then the code will use the default locations for Chrome and ChromeDriver.
  6. Build the solution and run!

This should open Chrome and automatically visit my example web page. The JavaScript in that web page makes two API calls, returning users and albums data as JSON in the HTTP responses. The code in CaptureResponse.cs captures the HTTP responses, stores the JSON from the message body, then outputs the JSON in the console window. In a real use case, you'll probably do something more exciting with the HTTP response!