Java+Selenium+Firefox proxy server cannot be set.
smatei opened this issue · 5 comments
Hi,
I have a problem configuring Firefox to use proxy from selenium and java. I used a similar example with the code snippet you provided here http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp
Here is the code I use:
`
import java.io.File;
import java.io.IOException;
import org.apache.commons.lang.SystemUtils;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxBinary;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.firefox.GeckoDriverService;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import com.google.common.collect.ImmutableMap;
public class TestProxy
{
public static void main(String[] args)
{
if (args.length == 0)
{
System.out.println("usage: IP port");
System.exit(-1);
}
String proxyServer = args[0];
int proxyPort = Integer.parseInt(args[1]);
try
{
TestProxy test = new TestProxy();
test.TestDriver(proxyServer, proxyPort);
} catch (IOException e)
{
e.printStackTrace();
}
}
private void TestDriver(String aHost, int aPort) throws IOException
{
FirefoxProfile firefoxProfile = new FirefoxProfile();
firefoxProfile.setEnableNativeEvents(true);
WebDriver driver = null;
GeckoDriverService service = null;
if (SystemUtils.IS_OS_LINUX)
{
FirefoxBinary firefoxBinary = new FirefoxBinary(new File("/home/ubuntu/firefox/firefox"));
service = new GeckoDriverService.Builder(firefoxBinary)
.usingDriverExecutable(new File("/usr/bin/geckodriver"))
.usingAnyFreePort()
.usingAnyFreePort()
.withEnvironment(ImmutableMap.of("DISPLAY",":20"))
.build();
service.start();
}
else if (SystemUtils.IS_OS_WINDOWS)
{
FirefoxBinary firefoxBinary = new FirefoxBinary(new File("C:\\Program Files\\Mozilla Firefox\\firefox.exe"));
service = new GeckoDriverService.Builder(firefoxBinary)
.usingDriverExecutable(new File("C:\\Windows\\geckodriver.exe"))
.usingAnyFreePort()
.usingAnyFreePort()
.build();
service.start();
}
else
{
System.out.println("Unknown operation system");
System.exit(-1);
}
DesiredCapabilities cap = new DesiredCapabilities();
Proxy proxy = new Proxy();
String proxyServer = aHost + ":" + aPort;
proxy.setHttpProxy(proxyServer)
.setFtpProxy(proxyServer)
.setSslProxy(proxyServer);
proxy.setProxyType(ProxyType.MANUAL);
cap.setCapability(CapabilityType.PROXY, proxy);
driver = new FirefoxDriver(service, cap, null);
driver.navigate().to("https://api.ipify.org/?format=text");
System.out.println(driver.getPageSource());
}
}
`
Here are the environment details, on both windows and linux:
Selenium version: client-combined-3.0.0-beta4-nodeps.jar
Firefox version: Mozilla Firefox 49.0.1 (both on Windows and Linux)
Windows version: 8.1
Linux version: Ubuntu 15.04
geckodriver version: 0.11.1 (on both Windows and Linux)
java version: build 1.8.0_102-b14 on 64 bits (on both Windows and Linux)
The output on Windows is:
1476874782195 geckodriver INFO Listening on 127.0.0.1:14369 Oct 19, 2016 1:59:42 PM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Attempting bi-dialect session, assuming Postel's Law holds true on the remote end log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies). log4j:WARN Please initialize the log4j system properly. 1476874783017 mozprofile::profile INFO Using profile path C:\Users\Stefan\AppData\Local\Temp\rust_mozprofile.j9tzNXSwtgD6 1476874783019 geckodriver::marionette INFO Starting browser C:\Program Files\Mozilla Firefox\firefox.exe 1476874783021 geckodriver::marionette INFO Connecting to Marionette on localhost:60470 1476874783502 Marionette INFO Listening on port 60470 1476874784363 Marionette INFO startBrowser 09aa9da8-d090-42af-be6d-3eec1ec1d4a2 Oct 19, 2016 1:59:44 PM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: W3C <html><head><link rel="alternate stylesheet" type="text/css" href="resource://gre-resources/plaintext.css" title="Wrap Long Lines"></head><body><pre>8x.7x.22x.14x</pre></body></html>
Where 8x.7x.22x.14x is my local ip, not proxy IP.
Similar output on Linux:
1476875492936 geckodriver INFO Listening on 127.0.0.1:22032 Oct 19, 2016 11:11:33 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Attempting bi-dialect session, assuming Postel's Law holds true on the remote end log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies). log4j:WARN Please initialize the log4j system properly. 1476875493387 mozprofile::profile INFO Using profile path /tmp/rust_mozprofile.9tk1zKafY0lI 1476875493387 geckodriver::marionette INFO Starting browser /home/ubuntu/firefox/firefox 1476875493389 geckodriver::marionette INFO Connecting to Marionette on localhost:38328 (firefox:25100): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed (firefox:25100): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed (firefox:25100): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed (firefox:25100): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed 1476875494041 Marionette INFO Listening on port 38328 1476875495209 Marionette INFO startBrowser 1d7fe1dd-c381-4287-852d-1a2d87dc324e Oct 19, 2016 11:11:35 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: W3C (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed (/home/ubuntu/firefox/plugin-container:25163): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed <html><head><link rel="alternate stylesheet" type="text/css" href="resource://gre-resources/plaintext.css" title="Wrap Long Lines"></head><body><pre>5x.14x.2x.2xx</pre></body></html>
Where 5x.14x.2x.2xx is the ip of the Linux machine not the proxy.
Where is the problem? If I open the firefox "Connection Settings", the proxy option is "Use System Proxy Settings", not manual.
I have also tried a version with FirefoxProfile preferences, but no luck either.
firefoxProfile.setPreference("network.proxy.type", 1); firefoxProfile.setPreference("network.proxy.http", proxyServer); firefoxProfile.setPreference("network.proxy.http_port", proxyPort); firefoxProfile.setPreference("network.proxy.ssl", proxyServer); firefoxProfile.setPreference("network.proxy.ssl_port", proxyPort); firefoxProfile.setPreference("network.proxy.ftp", proxyServer); firefoxProfile.setPreference("network.proxy.ftp_port", proxyPort); firefoxProfile.setPreference("network.proxy.socks", proxyServer); firefoxProfile.setPreference("network.proxy.socks_port", proxyPort); firefoxProfile.setPreference("network.proxy.share_proxy_settings", true); firefoxProfile.setPreference("network.proxy.no_proxies_on", "localhost, 127.0.0.1");
duplicate of mozilla/geckodriver#97
Hi @lukeis,
I have looked here mozilla/geckodriver#97
and in the geckodriver code
` GeckoDriver.prototype.setSessionCapabilities = function(newCaps) {
const copy = (from, to={}) => {
let errors = [];
// Remove any duplicates between required and desired in favour of the
// required capabilities
if (from !== null && from.desiredCapabilities) {
for (let cap in from.requiredCapabilities) {
if (from.desiredCapabilities[cap]) {
delete from.desiredCapabilities[cap];
}
}
// Let's remove the sessionCapabilities from desired capabilities
for (let cap in this.sessionCapabilities) {
if (from.desiredCapabilities && from.desiredCapabilities[cap]) {
delete from.desiredCapabilities[cap];
}
}
}
for (let key in from) {
switch (key) {
case "desiredCapabilities":
to = copy(from[key], to);
break;
case "requiredCapabilities":
if (from[key].proxy) {
this.setUpProxy(from[key].proxy);
to.proxy = from[key].proxy;
delete from[key].proxy;
}
for (let caps in from[key]) {
if (from[key][caps] !== this.sessionCapabilities[caps]) {
errors.push(from[key][caps] + " does not equal " +
this.sessionCapabilities[caps]);
}
}
break;
default:
to[key] = from[key];
}
}
if (Object.keys(errors).length == 0) {
return to;
}
throw new SessionNotCreatedError(
`Not all requiredCapabilities could be met: ${JSON.stringify(errors)}`);
};
// clone, overwrite, and set
let caps = copy(this.sessionCapabilities);
caps = copy(newCaps, caps);
logger.config("Changing capabilities: " + JSON.stringify(caps));
this.sessionCapabilities = caps;
};
`
It seems that it is looking for proxy only in 'requiredCapabilities'
` GeckoDriver.prototype.setUpProxy = function(proxy) {
logger.config("User-provided proxy settings: " + JSON.stringify(proxy));
if (typeof proxy == "object" && proxy.hasOwnProperty("proxyType")) {
switch (proxy.proxyType.toUpperCase()) {
case "MANUAL":
Preferences.set("network.proxy.type", 1);
if (proxy.httpProxy && proxy.httpProxyPort){
Preferences.set("network.proxy.http", proxy.httpProxy);
Preferences.set("network.proxy.http_port", proxy.httpProxyPort);
}
if (proxy.sslProxy && proxy.sslProxyPort){
Preferences.set("network.proxy.ssl", proxy.sslProxy);
Preferences.set("network.proxy.ssl_port", proxy.sslProxyPort);
}
if (proxy.ftpProxy && proxy.ftpProxyPort) {
Preferences.set("network.proxy.ftp", proxy.ftpProxy);
Preferences.set("network.proxy.ftp_port", proxy.ftpProxyPort);
}
if (proxy.socksProxy) {
Preferences.set("network.proxy.socks", proxy.socksProxy);
Preferences.set("network.proxy.socks_port", proxy.socksProxyPort);
if (proxy.socksVersion) {
Preferences.set("network.proxy.socks_version", proxy.socksVersion);
}
}
break;
case "PAC":
Preferences.set("network.proxy.type", 2);
Preferences.set("network.proxy.autoconfig_url", proxy.proxyAutoconfigUrl);
break;
case "AUTODETECT":
Preferences.set("network.proxy.type", 4);
break;
case "SYSTEM":
Preferences.set("network.proxy.type", 5);
break;
case "NOPROXY":
default:
Preferences.set("network.proxy.type", 0);
break;
}
} else {
throw new InvalidArgumentError("Value of 'proxy' should be an object");
}
};`
So I have tried to pass them like this:
` DesiredCapabilities required = new DesiredCapabilities();
JsonObject json = new JsonObject();
json.addProperty("proxyType", "MANUAL");
json.addProperty("httpProxy", aHost);
json.addProperty("httpProxyPort", aPort);
json.addProperty("sslProxy", aHost);
json.addProperty("sslProxyPort", aPort);
required.setCapability("proxy", json.toString());
driver = new FirefoxDriver(service, cap, required);
`
but still no luck, I get some error.
{"error":"invalid argument","message":"Value of 'proxy' should be an object","stacktrace":""},null]
I will try to find out why.
Hi @lukeis,
Success! I was sending:
setCapability("proxy", json.toString());
Instead of this now I use successfully
setCapability("proxy", json);
That opens firefox asking me for the user and password for that proxy.
` DesiredCapabilities required = new DesiredCapabilities();
JsonObject json = new JsonObject();
json.addProperty("proxyType", "MANUAL");
json.addProperty("httpProxy", aHost);
json.addProperty("httpProxyPort", aPort);
json.addProperty("sslProxy", aHost);
json.addProperty("sslProxyPort", aPort);
required.setCapability("proxy", json);
driver = new FirefoxDriver(service, cap, required);
`
Now we have the prove that proxy works with Marionette and Firefox.
You should get rid of the webdriver example
` String PROXY = "localhost:8080";
org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy(PROXY)
.setFtpProxy(PROXY)
.setSslProxy(PROXY);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, proxy);
WebDriver driver = new FirefoxDriver(cap);`
or at least mark it as deprecated and add the new example with Marionette desiredCapabilities.
Feel free to submit a patch to the website itself, I would appreciate it :)
Work in Selenium 3.14.2, Firefox 62, C# .NET 4.5
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(@"GeckoDriver19", "geckodriver.exe");
service.FirefoxBinaryPath = @"C:\Program Files\Mozilla Firefox\firefox.exe";
FirefoxOptions firefoxOptions = new FirefoxOptions();
firefoxOptions.SetPreference("network.proxy.type", 1);
firefoxOptions.SetPreference("network.proxy.socks", "127.0.0.1");
firefoxOptions.SetPreference("network.proxy.socks_port", 1080);
IWebDriver driver = new FirefoxDriver(service, firefoxOptions);
driver.Navigate().GoToUrl("https://www.hbus.com/register");