Is there a way to start chrome in the background?

[NatanFreeman]

I am new to using chromedriver and am using this crate to control Google Chrome. I just was wondering if it was possible to run Chrome silently without displaying a new window. There seem to be ways to do this in other languages, does this crate support doing so as well? Any help would be much appreciated.

[bcpeinhardt]

I think you're referring to running the browser in headless mode (so the browser isn't displayed when running). There is a method on each browser specific Capabilities struct to run headless called set_headless.

[NatanFreeman]

@bcpeinhardt thank you. set_headless does make it run without opening a window though it doesn't seem to load certain websites correctly. I used the screenshot command to capture the page for this website and it looks like this:

This is similar to what running curl gives you. When not running headless it looks normal. I think it's not running some JavaScript.

[bcpeinhardt]

Did you happen to take the screenshot right after loading the page? It may be that the page is "loaded" in the sense that there is html rendered to the browser but that there's a lot of JavaScript that needs to run before it looks normal. Try just waiting 10 seconds between loading the page and taking the screenshot to see if it looks right then we can look at changing your page load strategy it that is the issue.

[NatanFreeman]

Waiting 10 seconds didn't fix the issue. The website takes less time than that to load, so I doubt it's a time sensitive.

[bcpeinhardt]

Could you send me a link to the code or provide a code snippet I can run to recreate the issue?

[NatanFreeman]

Certainly! This example code works just fine when caps.set_headless()?; is emitted.

use thirtyfour::prelude::*;

#[tokio::main]
async fn main() -> WebDriverResult<()> {
    let mut caps = DesiredCapabilities::chrome();
    caps.set_headless()?;

    let driver = WebDriver::new("http://localhost:9515", caps).await?;

    driver.set_window_rect(0, 0, 1200, 900).await?;
    driver
        .goto("https://maya.tase.co.il/reports/breakingannouncement")
        .await?;

    let elem_form = driver
        .find_all(By::ClassName("feedItemDate.hidden-xs.hidden-sm.ng-binding"))
        .await?;

    for i in elem_form {
        println!("{i}");
    }

    driver.quit().await?;
    Ok(())
}

[bcpeinhardt]

Seems like the website may be detecting that you're using a headless browser and disabling some features. That might be because they don't want you scraping the site. Assuming you know they don't mind you scraping the site, this stack overflow question may be a good place to start: https://stackoverflow.com/questions/67617101/how-to-enable-javascript-with-headless-chrome-in-selenium.

As a small side note, you might consider using the query interface to find the elements you're looking for:

let elem_form = driver
        .query(By::ClassName("feedItemDate"))
        .and_displayed()
        .all()
        .await?;

Hope this helps!

[NatanFreeman]

I have been working on implementing this example in Rust. The main issue I have is this line:

driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": userAgent})

How would you generate a fake user and load it like this in Rust?

[NatanFreeman]

I'm closing this issue as the problem I'm having is unrelated to this crate specifically.