BrowserMob Proxy is a simple utility that makes it easy to capture performance data from browsers, typically written using automation toolkits such as Selenium and Watir.
The latest version of BrowserMobProxy is 2.1.1, powered by LittleProxy.
To use BrowserMob Proxy in your tests or application, add the browsermob-core dependency to your pom:
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.1</version>
<scope>test</scope>
</dependency>To run in standalone mode from the command line, download the latest release from the releases page, or build the latest from source.
For more information on using BrowserMob Proxy with Selenium, see the Using with Selenium section.
The browsermob-core-littleproxy module is now browsermob-core
After six beta releases, the LittleProxy implementation now supports more features and is more stable than the legacy implementation. To reflect that level of maturity and long-term support, the browsermob-core module now uses LittleProxy by default.
Note about Legacy support: In the 2.1-betas, if you were using the ProxyServer or LegacyProxyServer classes, use the browsermob-core-legacy module in 2.1.0 and higher.
LittleProxy support for LegacyProxyServer has moved to BrowserMobProxyServerLegacyAdapter. Using the LittleProxy implementation with the LegacyProxyServer interface is still fully supported as a means to help you transition from 2.0.0. Unlike the 2.1-beta series, the BrowserMobProxyServer class
no longer implements LegacyProxyServer; however, the BrowserMobProxyServerLegacyAdapter can be used to integrate legacy code with the new LittleProxy interface. You must still use the browsermob-core-legacy module when using the LegacyAdapter.
LegacyProxyServer proxy = new BrowserMobProxyServerLegacyAdapter();
proxy.setPort(8081); // method only supported by the legacy interface
proxy.start();The new BrowserMobProxyServer class has replaced the legacy ProxyServer implementation. The legacy implementation is no longer actively supported; all new code should use BrowserMobProxyServer. We highly recommend that existing code migrate to the new implementation.
The most important changes from 2.0 are:
- Separate REST API and Embedded Mode modules. Include only the functionality you need.
- New BrowserMobProxy interface. The new interface will completely replace the legacy 2.0 ProxyServer contract in version 3.0 and higher.
- LittleProxy support. More stable and more powerful than the legacy Jetty back-end.
BrowserMob Proxy 2.1 includes a new BrowserMobProxy interface to interact with BrowserMob Proxy programmatically. The new interface defines the functionality that BrowserMob Proxy will support in future releases (including 3.0+). To ease migration, both the legacy (Jetty-based) ProxyServer class and the new, LittleProxy-powered BrowserMobProxy class support the new BrowserMobProxy interface.
We highly recommend migrating existing code to the BrowserMobProxy interface using the BrowserMobProxyServer class.
The legacy interface, implicitly defined by the ProxyServer class, has been extracted into net.lightbody.bmp.proxy.LegacyProxyServer and is now officially deprecated. The new LittleProxy-based implementation will implement LegacyProxyServer for all 2.1.x releases. This means you can switch to the LittleProxy-powered implementation with minimal change to existing code (with the exception of interceptors):
// With the Jetty-based 2.0.0 release, BMP was created like this:
ProxyServer proxyServer = new ProxyServer();
proxyServer.start();
// [...]
// To use the LittleProxy-powered 2.1.1 release, simply change to
// the LegacyProxyServer interface and the adapter for the new
// LittleProxy-based implementation:
LegacyProxyServer proxyServer = new BrowserMobProxyServerLegacyAdapter();
proxyServer.start();
// Almost all deprecated 2.0.0 methods are supported by the
// new BrowserMobProxyServerLegacyAdapter implementation, so in most cases,
// no further code changes are necessaryLegacyProxyServer will not be supported after 3.0 is released, so we recommend migrating to the BrowserMobProxy interface as soon as possible. The new interface provides additional functionality and is compatible with both the legacy Jetty-based ProxyServer implementation (with some exceptions) and the new LittleProxy implementation.
If you must continue using the legacy Jetty-based implementation, include the browsermob-core-legacy artifact instead of browsermob-core.
The proxy is programmatically controlled via a REST interface or by being embedded directly inside Java-based programs and unit tests. It captures performance data in the HAR format. In addition it can actually control HTTP traffic, such as:
- blacklisting and whitelisting certain URL patterns
- simulating various bandwidth and latency
- remapping DNS lookups
- flushing DNS caching
- controlling DNS and request timeouts
- automatic BASIC authorization
New in 2.1: The REST API now supports LittleProxy. As of 2.1.0-beta-3, LittleProxy is the default implementation. (You may specify --use-littleproxy false to disable LittleProxy in favor of the legacy Jetty 5-based implementation.)
To get started, first start the proxy by running browsermob-proxy or browsermob-proxy.bat in the bin directory:
$ sh browsermob-proxy -port 9090
INFO 05/31 03:12:48 o.b.p.Main - Starting up...
2011-05-30 20:12:49.517:INFO::jetty-7.3.0.v20110203
2011-05-30 20:12:49.689:INFO::started o.e.j.s.ServletContextHandler{/,null}
2011-05-30 20:12:49.820:INFO::Started SelectChannelConnector@0.0.0.0:9090
Once started, there won't be an actual proxy running until you create a new proxy. You can do this by POSTing to /proxy:
[~]$ curl -X POST http://localhost:9090/proxy
{"port":9091}
or optionally specify your own port:
[~]$ curl -X POST -d 'port=9099' http://localhost:9090/proxy
{"port":9099}
or if running BrowserMob Proxy in a multi-homed environment, specify a desired bind address (default is 0.0.0.0):
[~]$ curl -X POST -d 'bindAddress=192.168.1.222' http://localhost:9090/proxy
{"port":9096}
Once that is done, a new proxy will be available on the port returned. All you have to do is point a browser to that proxy on that port and you should be able to browse the internet. The following additional APIs will then be available:
| Description | HTTP method | Request path | Request parameters |
|---|---|---|---|
Get a list of ports attached to ProxyServer instances managed by ProxyManager |
GET | /proxy | |
| Creates a new HAR attached to the proxy and returns the HAR content if there was a previous HAR. [port] in request path it is port where your proxy was started | PUT | /proxy/[port]/har | captureHeaders - Boolean, capture headers or not. Optional, default to "true". captureContent - Boolean, capture content bodies or not. Optional, default to "false". captureBinaryContent - Boolean, capture binary content or not. Optional, default to "false". initialPageRef - The string name of The first page ref that should be used in the HAR. Optional, default to "Page 1". initialPageTitle - The title of first HAR page. Optional, default to initialPageRef. |
| Starts a new page on the existing HAR. [port] in request path it is port where your proxy was started | PUT | /proxy/[port]/har/pageRef | pageRef - The string name of the first page ref that should be used in the HAR. Optional, default to "Page N" where N is the next page number. pageTitle - The title of new HAR page. Optional, default to |
| Shuts down the proxy and closes the port. [port] in request path it is port where your proxy was started | DELETE | /proxy/[port] | |
| Returns the JSON/HAR content representing all the HTTP traffic passed through the proxy (provided you have already created the HAR with this method) | GET | /proxy/[port]/har | |
| Displays whitelisted items | GET | /proxy/[port]/whitelist | |
| Sets a list of URL patterns to whitelist | PUT | /proxy/[port]/whitelist | regex - A comma separated list of regular expressions. status - The HTTP status code to return for URLs that do not match the whitelist. |
| Clears all URL patterns from the whitelist | DELETE | /proxy/[port]/whitelist | |
| Displays blacklisted items | GET | /proxy/[port]/blacklist | |
| Set a URL to blacklist | PUT | /proxy/[port]/blacklist | regex - The blacklist regular expression. status - The HTTP status code to return for URLs that are blacklisted. method - The regular expression for matching HTTP method (GET, POST, PUT, etc). Optional, by default processing all HTTP method. |
| Clears all URL patterns from the blacklist | DELETE | /proxy/[port]/blacklist | |
| Limit the bandwidth through the proxy on the [port] | PUT | /proxy/[port]/limit | downstreamKbps - Sets the downstream bandwidth limit in kbps. Optional. upstreamKbps - Sets the upstream bandwidth limit kbps. Optional, by default unlimited. upstreamKbps - Sets the upstream bandwidth limit kbps. Optional, by default unlimited. downstreamMaxKB - Specifies how many kilobytes in total the client is allowed to download through the proxy. Optional, by default unlimited. upstreamMaxKB - Specifies how many kilobytes in total the client is allowed to upload through the proxy. Optional, by default unlimited. latency - Add the given latency to each HTTP request. Optional, by default all requests are invoked without latency. enable - A boolean that enable bandwidth limiter. Optional, by default to "false", but setting any of the properties above will implicitly enable throttling payloadPercentage - Specifying what percentage of data sent is payload, e.g. use this to take into account overhead due to tcp/ip. Optional. maxBitsPerSecond - The max bits per seconds you want this instance of StreamManager to respect. Optional. |
| Displays the amount of data remaining to be uploaded/downloaded until the limit is reached | GET | /proxy/[port]/limit | |
| Set and override HTTP Request headers | POST | /proxy/[port]/headers | Payload data should be JSON encoded set of headers. Where key is a header name (such as "User-Agent") and value is a value of HTTP header to setup (such as "BrowserMob-Agent"). Example: {"User-Agent": "BrowserMob-Agent"} |
| Overrides normal DNS lookups and remaps the given hosts with the associated IP address | POST | /proxy/[port]/hosts | Payload data should be JSON encoded set of hosts. Where key is a host name (such as "example.com") and value is a IP address which associatied with host hame (such as "1.2.3.4"'). Example: {"example.com": "1.2.3.4"} |
| Sets automatic basic authentication for the specified domain | POST | /proxy/[port]/auth/basic/[domain] | Payload data should be JSON encoded username and password name/value pairs. Example: {"username": "myUsername", "password": "myPassword"} |
| Wait till all request are being made | PUT | /proxy/[port]/wait | quietPeriodInMs - Wait till all request are being made. Optional. timeoutInMs - Sets quiet period in milliseconds. Optional. |
| Handles different proxy timeouts | PUT | proxy/[port]/timeout | Payload data should be JSON encoded set of parameters. Where key is a parameters name (such as "connectionTimeout") and value is a value of parameter to setup (such as "500") requestTimeout - Request timeout in milliseconds. A timeout value of -1 is interpreted as infinite timeout. Optional, default to "-1". readTimeout - Read timeout in milliseconds. Which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets). A timeout value of zero is interpreted as an infinite timeout. Optional, default to "60000". connectionTimeout - Determines the timeout in milliseconds until a connection is established. A timeout value of zero is interpreted as an infinite timeout. Optional, default to "60000". dnsCacheTimeout - Sets the maximum length of time that records will be stored in this Cache. A nonpositive value disables this feature (that is, sets no limit). Optional, default to "0". Example:{"connectionTimeout" : "500", "readTimeout" : "200"} |
| Redirecting URL's | PUT | /proxy/[port]/rewrite | matchRegex - A matching URL regular expression. replace - replacement URL. |
| Removes all URL redirection rules currently in effect | DELETE | /proxy/[port]/rewrite | |
| Setting the retry count | PUT | /proxy/[port]/retry | retrycount - The number of times a method will be retried. |
| Empties the DNS cache | DELETE | /proxy/[port]/dns/cache | |
| REST API interceptors with LittleProxy | |||
| Describe your own request interception | POST | /proxy/[port]/filter/request | A string wich determinates interceptor rules. See more here |
| Describe your own response interception | POST | /proxy/[port]/filter/response | A string wich determinates interceptor rules. See more here |
| REST API with Legacy interceptors | |||
| Describe your own request interception | POST | /proxy/[port]/interceptor/request | A string wich determinates interceptor rules. See more here |
| Describe your own response interception | POST | /proxy/[port]/interceptor/response | A string wich determinates interceptor rules. See more here |
For example, once you've started the proxy you can create a new HAR to start recording data like so:
[~]$ curl -X PUT -d 'initialPageRef=Foo' http://localhost:8080/proxy/9091/har
Now when traffic goes through port 9091 it will be attached to a page reference named "Foo". Consult the HAR specification for more info on what a "pageRef" is. You can also start a new pageRef like so:
[~]$ curl -X PUT -d 'pageRef=Bar' http://localhost:8080/proxy/9091/har/pageRef
That will ensure no more HTTP requests get attached to the old pageRef (Foo) and start getting attached to the new pageRef (Bar). After creating the HAR, you can get its content at any time like so:
[~]$ curl http://localhost:8080/proxy/9091/har
Sometimes you will want to route requests through an upstream proxy server. In this case specify your proxy server by adding the httpProxy parameter to your create proxy request:
[~]$ curl -X POST http://localhost:9090/proxy?httpProxy=yourproxyserver.com:8080
{"port":9091}
Alternatively, you can specify the upstream proxy config for all proxies created using the standard JVM system properties for HTTP proxies. Note that you can still override the default upstream proxy via the POST payload, but if you omit the payload the JVM system properties will be used to specify the upstream proxy.
TODO: Other REST APIs supporting all the BrowserMob Proxy features will be added soon.
- -port <port>
- Port on which the API listens. Default value is 8080.
- -address
- Address to which the API is bound. Default value is 0.0.0.0.
- -proxyPortRange <from>-<to>
- Range of ports reserved for proxies. Only applies if port parameter is not supplied in the POST request. Default values are <port>+1 to <port>+500+1.
- -ttl <seconds>
- Proxy will be automatically deleted after a specified time period. Off by default.
New in 2.1: New Embedded Mode module
New in 2.1: New BrowserMobProxy interface for Embedded Mode
BrowserMob Proxy 2.1 separates the Embedded Mode and REST API into two modules. If you only need Embedded Mode functionality, add the browsermob-core artifact as a dependency. The REST API artifact is browsermob-rest.
If you're using Java and Selenium, the easiest way to get started is to embed the project directly in your test. First, you'll need to make sure that all the dependencies are imported in to the project. You can find them in the lib directory. Or, if you're using Maven, you can add this to your pom:
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.1</version>
<scope>test</scope>
</dependency>Once done, you can start a proxy using net.lightbody.bmp.BrowserMobProxy:
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
// get the JVM-assigned port and get to work!
int port = proxy.getPort();
//...
Consult the Javadocs on the net.lightbody.bmp.BrowserMobProxy class for the full API.
You can use the REST API with Selenium however you want. But if you're writing your tests in Java and using Selenium 2, this is the easiest way to use it:
// start the proxy
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
// get the Selenium proxy object
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);
// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "yahoo.com"
proxy.newHar("yahoo.com");
// open yahoo.com
driver.get("http://yahoo.com");
// get the HAR data
Har har = proxy.getHar();HTTP request manipulation has changed in 2.1.0+ with LittleProxy. The LittleProxy-based interceptors are easier to use and more reliable. The legacy ProxyServer implementation will not support the new interceptor methods.
There are four new methods to support request and response interception in LittleProxy:
addRequestFilteraddResponseFilteraddFirstHttpFilterFactoryaddLastHttpFilterFactory
For most use cases, including inspecting and modifying requests/responses, addRequestFilter and addResponseFilter will be sufficient. The request and response filters are easy to use:
proxy.addRequestFilter(new RequestFilter() {
@Override
public HttpResponse filterRequest(HttpRequest request, HttpMessageContents contents, HttpMessageInfo messageInfo) {
if (messageInfo.getOriginalUri().endsWith("/some-endpoint-to-intercept")) {
// retrieve the existing message contents as a String or, for binary contents, as a byte[]
String messageContents = contents.getTextContents();
// do some manipulation of the contents
String newContents = messageContents.replaceAll("original-string", "my-modified-string");
//[...]
// replace the existing content by calling setTextContents() or setBinaryContents()
contents.setTextContents(newContents);
}
// in the request filter, you can return an HttpResponse object to "short-circuit" the request
return null;
}
});
// responses are equally as simple:
proxy.addResponseFilter(new ResponseFilter() {
@Override
public void filterResponse(HttpResponse response, HttpMessageContents contents, HttpMessageInfo messageInfo) {
if (/*...some filtering criteria...*/) {
contents.setTextContents("This message body will appear in all responses!");
}
}
});With Java 8, the syntax is even more concise:
proxy.addResponseFilter((response, contents, messageInfo) -> {
if (/*...some filtering criteria...*/) {
contents.setTextContents("This message body will appear in all responses!");
}
});See the javadoc for the RequestFilter and ResponseFilter classes for more information.
For fine-grained control over the request and response lifecycle, you can add "filter factories" directly using addFirstHttpFilterFactory and addLastHttpFilterFactory (see the examples in the InterceptorTest unit tests).
When running the REST API with LittleProxy enabled, you cannot use the legacy /:port/interceptor/ endpoints. Instead, POST the javascript payload to the new /:port/filter/request and /:port/filter/response endpoints.
Javascript request filters have access to the variables request (type io.netty.handler.codec.http.HttpRequest), contents (type net.lightbody.bmp.util.HttpMessageContents), and messageInfo (type net.lightbody.bmp.util.HttpMessageInfo). messageInfo contains additional information about the message, including whether the message is sent over HTTP or HTTPS, as well as the original request received from the client before any changes made by previous filters. If the javascript returns an object of type io.netty.handler.codec.http.HttpResponse, the HTTP request will "short-circuit" and return the response immediately.
Example: Modify User-Agent header
curl -i -X POST -H 'Content-Type: text/plain' -d "request.headers().remove('User-Agent'); request.headers().add('User-Agent', 'My-Custom-User-Agent-String 1.0');" http://localhost:8080/proxy/8081/filter/requestJavascript response filters have access to the variables response (type io.netty.handler.codec.http.HttpResponse), contents (type net.lightbody.bmp.util.HttpMessageContents), and messageInfo (type net.lightbody.bmp.util.HttpMessageInfo). As in the request filter, messageInfo contains additional information about the message.
Example: Modify response body
curl -i -X POST -H 'Content-Type: text/plain' -d "contents.setTextContents('<html><body>Response successfully intercepted</body></html>');" http://localhost:8080/proxy/8081/filter/responseIf you are using the legacy ProxyServer implementation, you can manipulate the requests like so:
BrowserMobProxy server = new ProxyServer();
((LegacyProxyServer)server).addRequestInterceptor(new RequestInterceptor() {
@Override
public void process(BrowserMobHttpRequest request, Har har) {
request.getMethod().removeHeaders("User-Agent");
request.getMethod().addHeader("User-Agent", "Bananabot/1.0");
}
});You can also POST a JavaScript payload to /:port/interceptor/request and /:port/interceptor/response using the REST interface. The functions will have a request/response variable, respectively, and a har variable (which may be null if a HAR isn't set up yet). The JavaScript code will be run by Rhino and have access to the same Java API in the example above:
[~]$ curl -X POST -H 'Content-Type: text/plain' -d 'request.getMethod().removeHeaders("User-Agent");' http://localhost:9090/proxy/9091/interceptor/request
Consult the Java API docs for more info.
BrowserMob Proxy 2.1.0+ now supports full MITM: For most users, MITM will work out-of-the-box with default settings. Install the ca-certificate-rsa.cer file in your browser or HTTP client to avoid untrusted certificate warnings. Generally, it is safer to generate your own private key, rather than using the .cer files distributed with BrowserMob Proxy. See the README file in the mitm module for instructions on generating or using your own root certificate and private key with MITM.
Legacy Jetty-based ProxyServer support for MITM: The legacy ProxyServer implementation uses the same ca-certificate-rsa.cer root certificate as the default BrowserMobProxyServer implementation. The previous cybervillainsCA.cer certificate has been removed.
Note: DO NOT permanently install the .cer files distributed with BrowserMob Proxy in users' browsers. They should be used for testing only and must not be used with general web browsing.
If you're doing testing with Selenium, you'll want to make sure that the browser profile that gets set up by Selenium not only has the proxy configured, but also has the CA installed. Unfortunately, there is no API for doing this in Selenium; it must be done manually for each browser and environment.
NodeJS bindings for browswermob-proxy are available here. Built-in support for Selenium or use CapserJS-on-PhantomJS or anything else to drive traffic for HAR generation.
When running in stand-alone mode, the proxy loads the default logging configuration from the conf/bmp-logging.yaml file. To increase/decrease the logging level, change the logging entry for net.lightbody.bmp.
The BrowserMobProxyServer implementation uses native DNS resolution by default, but supports custom DNS resolution and advanced DNS manipulation. See the ClientUtil class for information on DNS manipulation using the dnsjava resolver.
You'll need maven (brew install maven if you're on OS X); use the release profile to generate the batch files from this repository.
[~]$ mvn -DskipTests -P release
You'll find the standalone BrowserMob Proxy distributable zip at browsermob-dist/target/browsermob-proxy-2.1.1-SNAPSHOT-bin.zip. Unzip the contents and run the browsermob-proxy or browsermob-proxy.bat files in the bin directory.
When you build the latest code from source, you'll have access to the latest snapshot release. To use the SNAPSHOT version in your code, modify the version in your pom:
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.1-SNAPSHOT</version>
<scope>test</scope>
</dependency>