lemanschik/awesome

Documentation build details chromium.

Opened this issue · 0 comments

lemanschik commented

Updates: Headless

see: https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md#usage-as-a-c_library
status: nopr

Build as a C++ library

Headless Chromium can be built as a library for embedding into a C++ application. This approach is otherwise similar to controlling the browser over a DevTools connection, but it provides more customization points, e.g., for networking and mojo services.

Headless Example is a small sample application which demonstrates the use of the headless C++ API. It loads a web page and outputs the resulting DOM. To run it, first initialize a headless build configuration:

$ mkdir -p out/Debug
--
  | $ echo 'import("//build/args/headless.gn")' > out/Debug/args.gn
  | $ gn gen out/Debug
  |

Then build the example:

$ ninja -C out/Debug headless_example
  |

After the build completes, the example can be run with the following command:

$ out/Debug/headless_example https://www.google.com
  |

Headless Shell is a more capable headless application. For instance, it supports remote debugging with the DevTools protocol. To do this, start the application with an argument specifying the debugging port:

$ ninja -C out/Debug headless_shell
  | $ out/Debug/headless_shell --remote-debugging-port=9222 https://youtube.com
  |

Then navigate to http://localhost:9222 with your browser.

Usage as a C++ library

Headless Chromium can be built as a library for embedding into a C++ application. This approach is otherwise similar to controlling the browser over a DevTools connection, but it provides more customization points, e.g., for networking and mojo services.

Headless Example is a small sample application which demonstrates the use of the headless C++ API. It loads a web page and outputs the resulting DOM. To run it, first initialize a headless build configuration:

$ mkdir -p out/Debug
$ echo 'import("//build/args/headless.gn")' > out/Debug/args.gn
$ gn gen out/Debug

Then build the example:

$ ninja -C out/Debug headless_example

After the build completes, the example can be run with the following command:

$ out/Debug/headless_example https://www.google.com

Headless Shell is a more capable headless application. For instance, it supports remote debugging with the DevTools protocol. To do this, start the application with an argument specifying the debugging port:

$ ninja -C out/Debug headless_shell
$ out/Debug/headless_shell --remote-debugging-port=9222 https://youtube.com

Then navigate to http://localhost:9222 with your browser.

Embedder API

The embedder API allows developers to integrate the headless library into their application. The API provides default implementations for low level adaptation points such as networking and the run loop.

The main embedder API classes are:

  • HeadlessBrowser::Options::Builder - Defines the embedding options, e.g.:
    • SetMessagePump - Replaces the default base message pump. See base::MessagePump.
    • SetProxyServer - Configures an HTTP/HTTPS proxy server to be used for accessing the network.

Client/DevTools API

The headless client API is used to drive the browser and interact with loaded web pages. Its main classes are:

  • HeadlessBrowser - Represents the global headless browser instance.
  • HeadlessWebContents - Represents a single “tab” within the browser.
  • HeadlessDevToolsClient - Provides a C++ interface for inspecting and controlling a tab. The API functions corresponds to DevTools commands. See the client API documentation for more information.

Resources and Documentation

Mailing list: headless-dev@chromium.org

Bug tracker: Internals>Headless

File a new bug (bit.ly/2pP6SBb)

Retrospective

Runtime headless mode on Windows OS
dvallet@
May 26th, 2017
Public document
Tracking bug: crbug.com/686608

Headless Chrome has been supported in Linux since end of 2016 (see crbug.com/612904). After providing Linux support, we set the goal of supporting Headless Chrome in the other two desktop platforms: Mac OS and Windows.

This document is a retrospective of the changes needed to make headless run in Windows, as well as the common challenges of developing headless in Windows.
The goal is to help future developers when adding features to all platforms.
Overview
The main changes needed to support --headless and build of headless_shell are:
(Headless_shell) Remove/Change Posix specific code from headless crrev.com/2666503002
(Headless_shell) Add Windows sandbox to headless crrev.com/2666503002
(Chrome --headless) Pass on sandbox info to Headless http://crrev.com/2762593002
(Chrome --headless) Fix Windows linker for Chrome target crrev.com/2775693003
(Chrome --headless) Refractor headless_lib so it can be linked in chrome:main_dll and chrome:child_dll crrev.com/276259300
(Chrome --headless) Avoid size regressions in Windows build. crrev.com/2887033003
Add crashpad support crrev.com/2835913002
Detailed changes
Remove/Change Posix specific code from headless (crrev.com/2666503002)
Change net:FileStream to base:FileProxy, since the former is posix only.
File paths need to be defined with the FILE_PATH_LITERAL macro.
Modify client_api_generator to avoid conflicting with windows only defined macros.
Change all command line calls to base::CommandLine::ForCurrentProcess().
Always enable logging in Windows
Add Windows sandbox initialization (crrev.com/2666503002)
build/win:default_exe_manifest needs to be added as a dependency to any windows executable that uses the sandbox. Otherwise, you get runtime errors when trying to obtain the Windows version.
Pass on sandbox info to Headless (http://crrev.com/2762593002)
HeadlessShellMain in windows needs to receive a valid, initialized sandbox. For chrome, this has to be passed onto headless, otherwise you get a render crash.
Windows linker errors (crrev.com/2775693003).
HEADLESS_EXPORT wasn’t properly used for Windows linking. The linker expects classes/structs, enums, and functions, to be properly declared as a dllexport or dllimport when they are used by an external component/library.

HEADLESS_EXPORT needs to be added when:
They are declared in a headless_implementation component/library.
They are used by another library/executable with no headless_implementation config., i.e.., headless_shell_lib, headless_shell_child_lib, headless_shell_browser_lib headless_shell.
Headless_lib refactoring (crrev.com/2762593002)
This refactoring was needed because chrome static libraries in Windows are split into two DLLs: chrome.dll and chrome_child.dll (check http://crbug.com/237249 for context).
chrome.dll contains only dependencies related to the main browser thread, while chrome_child.ll contains only dependency related to child processes (mostly renderer related).

The main changes needed were the following:
Modify headless_lib to be a component rather than a static library. This makes it easier to allow component and non_component builds in Windows, but also allows being able to reuse the core parts of headless as a component.
Add headless_shell_browser_lib and headless_shell_child_lib as specific libraries for chrome_main and chrome_child, respectively
Non-Component builds are defined to use chrome_main and chrome_child DLLs. This means that if there is any specific code that uses browser/renderer only dependencies it has to follow these rules (see crrev.com/2887033003 for an example):
It cannot be included in the headless component
It must be added in headless_shell_lib (for all platforms) and into the headless_shell_browser_lib and headless_shell_child_lib (Windows main/browser and child specific libraries, respectively).
In case of child specific classes, also include them in headless_renderer for browser and unit tests.
Use CHROME_MULTIPLE_DLL_BROWSER and CHROME_MULTIPLE_DLL_CHILD defines include specific main and child code/dependencies, respectively.
Component builds do need to include all browser/renderer specific code into the headless component (see headless/BUILD.gn), otherwise you’ll get linker errors.
Avoid size regressions in Windows build (crrev.com/2887033003).
The first time --headless landed it had to be reverted due to a size regression in chrome_child.dll (crbug.com/714841 and crbug.com/714888).
The main source of size regressions comes from adding dependencies to the renderer that are not needed. Note that, in this case, just removing the dependencies in the build file is not good enough, but also all references to HeadlesBrowserImpl and HeadlessWebContentsImpl had to be removed so that the linker can dispose of unused code appropriately. In order to do this we needed to:
Create windows versions of headless_shell_win.cc and headless_content_main_delegate_win.cc
Remove all references to HeadlessWebContents and BrowserMainRunner by using the CHROME_MULTIPLE_DLL_CHILD defines
Add crash reporting support (crrev.com/2835913002).
Linux uses breakpad as a crash reporter, while Mac and Windows use crashpad.
chrome --headless already captures crash reports, storing them in the default directory
For headless_shell, we have to fork the crashpad_handler and add the necessary dependencies to do this.
Common errors
Unused variables
VS compiler can produce compile errors due to non initialized variables, e.g. crrev.com/2666503002

Enums
error C2059: syntax error: 'constant'
This error is typically due to a conflict on constants defined only in windows, (e.g. NO_ERROR). It is advised to use class enums, but otherwise, changing the name of the enum should suffice.
Clang build in Windows
Note that clang compilation in Windows can generate different errors than in linux. Sometimes these are FYI build bots, so there are no try bots failing. See http://crrev.com/2878253002 for an example
HEADLESS_EXPORT
warning C4275: non dll-interface class 'headless::HeadlessBrowser' used as base for dll-interface class 'headless::HeadlessBrowserImpl'
This class needs a HEADLESS_EXPORT header. This also applies to enums/structs that are implemented in the headless component and used in headless_shell_(child|browser)_lib
Windows dll size troubleshooting (http://crbug.com/714841)
Why does it happen?
Windows splits DLLs into chrome_main and chrome_child DLLs. In Chrome, this usually means that main related code should only depend in content/public/app:browser, while childs’s should only depend on //content/public/app:child, with shared dependencies defined in content/public/app:both.

There are dependency restrictions set in chrome/BUILD.gn that will disallow render related dependencies in chrome_main. There is however no restriction on dependencies added to child processes.

In headless there is not that clear distinction between browser/child dependencies, so it’s relatively easy to add unnecessary deps to child that can cause increase in sizes in the chrome_child.dll object. This will cause an alert in chromeperf size bots and your CL to be likely reverted (see http://crbug.com/722400)
How to debug/reproduce an issue
Your best bet is to do a Windows non component build and compare the chrome_child.dll size to the a clean build. You should use the following args
is_component_build = false
is_debug = false
use_goma = true
target_cpu = "x64"

An increase on size here will most times translate to a similar increase in size on the official build, but note that this is not always accurate since official builds use WPO (Whole Program Optimization), which you can’t use in your local builds.
My DLL increase in size significantly, how can I solve/debug this?
This one is the tricky part, I’ll summarize the steps taken in http://crbug.com/714841. But I recommend following the bug to get a better sense of what’s needed. Most of the tools mentioned here are explained at https://www.chromium.org/developers/windows-binary-sizes.

SymbolSort
This is a useful tool to compare dll sizes. In my case, I had to do an official build to get data that made sense (and ultimately led to the solution). These are the arguments needed.
is_chrome_branded = true #Optional (only google internal)
is_debug = false
is_official_build = true
Check https://bugs.chromium.org/p/chromium/issues/detail?id=714841#c26 for more details on how to run and interpret the output. The goal is to find a specific symbol that has added a significant increase in size to the DLL and use the verbose linker output to figure out what’s pulling that symbol.
Generate a verbose linker output
You need to add “/VERBOSE” to the ldfags in chrome/BUILD/gn and do a non-component build. In this step you can use goma and it doesn’t need to be an official build.
Use linker verbose tracking tool
(tools\win\linker_verbose_tracking.py) to figure out what dependency is pulling the symbol found in step 1.
An example of this https://bugs.chromium.org/p/chromium/issues/detail?id=714841#c37
In short, you want to find a direct pulled dependency and make any sense of it. For instance, in the bug example, there was a symbol that was ultimately pulled by web_contents_impl.obj. Since WebContents is only used by the browser thread, this gave the clue it should be removed from the child library.
Removing a direct dependency
Your goal is to modify the code so that you don’t see that direct dependency any more (see https://bugs.chromium.org/p/chromium/issues/detail?id=714841#c42). For that you can use the CHROME_MULTIPLE_DLL_CHILD and CHROME_MULTIPLE_DLL_BROWSER defines, to def out code that shouldn’t be included in the child. See crrev/2887033003/ for this example.
Check results
You’ll probably see a decrease in size and in this it removed the size regression. If it is not enough, go back to step 1
Impact on binary size
The absolute increase sizes where
~160KB chrome.dll
~70KB for chrome_child.dll
~30KB mini_installer.exe
Full report here