Selenium Wire extends Selenium's Python bindings to give your tests access to the underlying requests made by the browser. It is a lightweight library designed for ease of use with minimal external dependencies.
With Selenium Wire, you author your tests in just the same way as you do with Selenium, but you get an additional user-friendly API for accessing things such as the request/response headers, status code and body content.
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# Go to the Google home page
driver.get('https://www.google.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.path,
request.response.status_code,
request.response.headers['Content-Type']
)
Prints:
https://www.google.com/ 200 text/html; charset=UTF-8
https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_120x44dp.png 200 image/png
https://consent.google.com/status?continue=https://www.google.com&pc=s×tamp=1531511954&gl=GB 204 text/html; charset=utf-8
https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png 200 image/png
https://ssl.gstatic.com/gb/images/i2_2ec824b0.png 200 image/png
https://www.google.com/gen_204?s=webaft&t=aft&atyp=csi&ei=kgRJW7DBONKTlwTK77wQ&rt=wsrt.366,aft.58,prt.58 204 text/html; charset=UTF-8
...
- Straightforward, user-friendly API
- All HTTP/HTTPS requests captured
- Access to request/response bodies
- Modify responses
- Header injection/filtering
- URL rewriting
- Proxy server support
- Python 3.4+
- Selenium 3.4.0+
- Firefox, Chrome, Safari and Edge are supported
- Installation
- Usage
- Request Attributes
- Response Attributes
- Modifying Requests
- Proxies
- Other Options
- Limitations
- License
Install using pip:
pip install selenium-wire
Selenium Wire requires OpenSSL for capturing HTTPS requests.
Linux
# For apt based Linux systems
sudo apt install openssl
# For RPM based Linux systems
sudo yum install openssl
MacOS
brew install openssl
Windows
No installation is required. OpenSSL for Windows is bundled with Selenium Wire.
Firefox and Chrome
No specific configuration should be necessary - everything should just work.
You will however need to ensure that you have downloaded the Gecko driver and Chrome driver for Firefox and Chrome to be remotely controlled - the same as if you were using Selenium directly. Once downloaded, these executables should be placed somewhere on the system path.
Safari
There are a few manual steps that have to be carried out before you can use Safari with Selenium Wire.
Edge
Like Safari, Microsoft Edge requires some manual configuration before it can be used with Selenium Wire.
Ensure that you import webdriver
from the seleniumwire
package:
from seleniumwire import webdriver
- For sub-packages of
webdriver
, you should continue to import these directly fromselenium
. For example, to importWebDriverWait
:
# Sub-packages of webdriver must still be imported from `selenium` itself
from selenium.webdriver.support.ui import WebDriverWait
For Firefox and Chrome, you don't need to do anything special. Just instantiate the webdriver as you would normally, passing in Selenium specific options if you have any. Selenium Wire also has it's own options that can be passed in the seleniumwire_options
attribute.
Firefox
driver = webdriver.Firefox()
Chrome
driver = webdriver.Chrome()
Safari
For Safari, you need to tell Selenium Wire the port number you selected when you configured the browser in Browser Setup.
For example, if you chose port 12345, then you would pass it in the seleniumwire_options
like this:
driver = webdriver.Safari(seleniumwire_options={'port': 12345})
Edge
For Edge, you need to tell Selenium Wire the port number you selected when you configured the browser in Browser Setup.
For example, if you chose port 12345, then you would pass it in the seleniumwire_options
like this:
driver = webdriver.Edge(seleniumwire_options={'port': 12345})
If the site you are testing uses a self-signed certificate then you must set the verify_ssl
option to False
in the seleniumwire_options
:
driver = webdriver.Firefox(seleniumwire_options={'verify_ssl': False})
This this will need to be done regardless of the type of browser you are using.
Selenium Wire captures all HTTP/HTTPS traffic made by the browser during a test.
driver.requests
You can retrieve all requests with the driver.requests
attribute. The requests are just a list and can be iterated (like in the opening example) and indexed:
first_request = driver.requests[0]
driver.last_request
The list of requests held by driver.requests
is in chronological order. If you want to access the most recent request, use the dedicated driver.last_request
attribute:
last_request = driver.last_request
This is more efficient than using driver.requests[-1]
.
When you ask for captured requests using driver.requests
or driver.last_request
you have to be sure that the requests you're interested in have actually been captured. If you ask too soon, then you may find that a request is not yet present, or is present but has no associated response.
driver.wait_for_request()
This method will wait for a previous request with a specific path to complete before allowing the test to continue. The path can be a unique part of the URL or the full URL itself.
For example, to wait for an AJAX request to return after a button is clicked:
# Click a button that triggers a background request to https://server/api/products/12345/
button_element.click()
# Wait for the request/response to complete
request = driver.wait_for_request('/api/products/12345/')
- Note that
driver.wait_for_request()
doesn't make a request, it just waits for a previous request made by some other action.
The wait_for_request()
method will return the first fully completed request it finds that matches the supplied path. Fully completed meaning that the response must have returned. The method will wait up to 10 seconds by default but you can vary that with the timeout
argument:
# Wait up to 30 seconds for a request/response
request = driver.wait_for_request('/api/products/12345/', timeout=30)
If a fully completed request is not seen within the timeout period a TimeoutException
is raised.
To clear previously captured requests, use del
:
del driver.requests
By default, Selenium Wire will capture all requests the browser makes during a test. You may want to restrict this to particular URLs - e.g. for performance reasons.
To restrict request capture use the scopes
attribute. This accepts a list of regular expressions that will match URLs to be captured.
driver.scopes = [
'.*stackoverflow.*',
'.*github.*'
]
# Only request URLs containing "stackoverflow" or "github" will now be captured...
Requests have the following attributes.
method
- The HTTP method type such as
GET
orPOST
. path
- The request path.
headers
- A case-insensitive dictionary of request headers. Asking for
request.headers['user-agent']
will return the value of theUser-Agent
header. body
- The request body as
bytes
. If the request has no body the value ofbody
will beNone
. response
- The response associated with the request. This will be
None
if the request has no response.
The response can be retrieved from a request via the response
attribute. A response may be None
if it was never captured, which may happen if you asked for it before it returned or if the server timed out etc. A response has the following attributes.
status_code
- The status code of the response such as
200
or404
. reason
- The reason phrase such as
OK
orNot Found
. headers
- A case-insensitive dictionary of response headers. Asking for
response.headers['content-length']
will return the value of theContent-Length
header. body
- The response body as
bytes
. If the response has no body the value ofbody
will beNone
.
Selenium Wire allows you to modify the request headers the browser sends as well as rewrite any part of the request URL.
The driver.header_overrides
attribute is used for modifying headers.
To add one or more new headers to a request, create a dictionary containing those headers and set it as the value of header_overrides
.
driver.header_overrides = {
'New-Header1': 'Some Value',
'New-Header2': 'Some Value'
}
# All subsequent requests will now contain New-Header1 and New-Header2
If a header already exists in a request it will be overwritten by the one in the dictionary. Header names are case-insensitive.
To filter out one or more headers from a request, set the value of those headers to None
.
driver.header_overrides = {
'Existing-Header1': None,
'Existing-Header2': None
}
# All subsequent requests will now *not* contain Existing-Header1 or Existing-Header2
To clear the header overrides that you have set, use del
:
del driver.header_overrides
Header overrides can also be applied on a per-URL basis, in the following format:
driver.header_overrides = [
('.*prod1.server.com.*', {'User-Agent': 'Test_User_Agent_String',
'New-Header': 'HeaderValue'}),
('.*prod2.server.com.*', {'User-Agent2': 'Test_User_Agent_String2',
'New-Header2': 'HeaderValue'})
]
# Only requests to prod1.server.com or prod2.server.com will have their headers modified
The driver.rewrite_rules
attribute is used for rewriting request URLs.
Each rewrite rule should be specified as a 2-tuple or list, the first element containing the URL pattern to match and the second element the replacement. One or more rewrite rules can be supplied.
driver.rewrite_rules = [
(r'(https?://)prod1.server.com(.*)', r'\1prod2.server.com\2'),
]
# All subsequent requests that match http://prod1.server.com... or https://prod1.server.com...
# will be rewritten to http://prod2.server.com... or https://prod2.server.com...
The match and replacement syntax is just Python's regex syntax. See re.sub for more information.
To clear the rewrite rules that you have set, use del
:
del driver.rewrite_rules
Selenium Wire captures requests by using its own proxy server under the covers. This means you cannot use the webdriver's DesiredCapabilities
API to configure your own proxy, like you might when using Selenium directly.
If the site you are testing sits behind a proxy server you can tell Selenium Wire about that proxy server in the options you pass to the webdriver instance. The configuration takes the following format:
options = {
'proxy': {
'http': 'http://username:password@host:port',
'https': 'https://username:password@host:port',
'no_proxy': 'localhost,127.0.0.1,dev_server:8080'
}
}
driver = webdriver.Firefox(seleniumwire_options=options)
The username and password are optional and can be specified when a proxy server requires authentication. Basic authentication is assumed by default.
The proxy configuration can also be loaded through environment variables called http
, https
and no_proxy
. The proxy configuration in the options passed to the webdriver instance will take precedence over environment variable configuration if both are specified.
Proxy authentication other than Basic
Basic authentication is used by default when supplying a username and password in the URL. If you are connecting to an upstream proxy server that uses an authentication scheme different to Basic, then you can supply the full value for the Proxy-Authorization
header using the custom_authorization
option. For example, if your proxy used the Bearer scheme:
options = {
'proxy': {
'http': 'http://host:port',
'https': 'https://host:port',
'no_proxy': 'localhost,127.0.0.1,dev_server:8080',
'custom_authorization': 'Bearer mytoken123' # Custom Proxy-Authorization header value
}
}
driver = webdriver.Firefox(seleniumwire_options=options)
Other options that can be passed to Selenium Wire via the seleniumwire_options
webdriver attribute:
connection_timeout
- The number of seconds Selenium Wire should wait before timing out requests. The default is 5 seconds. Increase this value if you're working with a slow server that needs more time to respond. Set to
None
for no timeout.
options = {
'connection_timeout': None # Never timeout
}
driver = webdriver.Firefox(seleniumwire_options=options)
custom_response_handler
- This function that should be passed in custom response handlers should maintain a signature that it compatible with
CaptureRequestHandler.response_handler
, as all arguments passed to that function will in turn be passed to your function. In order to modify the response data, you will need to return it from your function (the response data for the request is given in theres_body
argument).
def custom(req, req_body, res, res_body):
print(f'res_body length: {len(res_body)}')
options = {
'custom_response_handler': custom
}
drv = webdriver.Firefox(seleniumwire_options=options)
drv.get('https://example.com')
The code above will print something like this to the console (loading a page will almost always initiate more than one request):
res_body length: 471
res_body length: 606
ignore_http_methods
- A list of HTTP methods (specified as uppercase strings) that should be ignored by Selenium Wire and not captured. The default is
['OPTIONS']
which ignores all OPTIONS requests. To capture all request methods, setignore_http_methods
to an empty list:
options = {
'ignore_http_methods': [] # Capture all requests, including OPTIONS requests
}
driver = webdriver.Firefox(seleniumwire_options=options)
disable_encoding
- Whether to disable content encoding. When set to
True
, theAccept-Encoding
header will be set toidentity
for all requests. This tells the server to not compress/modify the response. The default isFalse
.
options = {
'disable_encoding': True # Tell the server not to compress the response
}
driver = webdriver.Firefox(seleniumwire_options=options)
suppress_connection_errors
- Whether to suppress connection related tracebacks. The default is
True
so that harmless errors that commonly occur at browser shutdown do not alarm users. When suppressed, the connection error message is logged at DEBUG level without a traceback. Set toFalse
to allow exception propagation and see full tracebacks.
options = {
'suppress_connection_errors': False # Show full tracebacks for any connection errors
}
driver = webdriver.Firefox(seleniumwire_options=options)
- Selenium Wire will currently work with tests that run on the same machine as the browser. A distributed setup using Selenium Grid is not yet supported.
- Sites that use NTLM authentication (Windows authentication) cannot currently be tested with Selenium Wire. NTLM authentication is not supported.
MIT