/rod

A Devtools driver for web automation and scraping

Primary LanguageGoMIT LicenseMIT

Overview

Go Reference Discord Chat

Documentation | API reference

Rod is a high-level driver directly based on DevTools Protocol. It's designed for web automation and scraping. Rod is designed for both high-level and low-level use, senior programmers can use the low-level packages and functions to easily customize or build up their own version of Rod, the high-level functions are just examples to build a default version of Rod.

Features

  • Chained context design, intuitive to timeout or cancel the long-running task
  • Auto-wait elements to be ready
  • Debugging friendly, auto input tracing, remote monitoring headless browser
  • Thread-safe for all operations
  • Automatically find or download browser
  • Lightweight, no third-party dependencies, CI tested on Linux, Mac, and Windows
  • High-level helpers like WaitStable, WaitRequestIdle, HijackRequests, WaitDownload, etc
  • Two-step WaitEvent design, never miss an event (how it works)
  • Correctly handles nested iframes or shadow DOMs
  • No zombie browser process after the crash (how it works)

Examples

Please check the examples_test.go file first, then check the examples folder.

For more detailed examples, please search the unit tests. Such as the usage of method HandleAuth, you can search all the *_test.go files that contain HandleAuth, for example, use Github online search in repository. You can also search the GitHub issues, they contain a lot of usage examples too.

Here is a comparison of the examples between rod and Chromedp.

If you have questions, please raise an issue or join the chat room.

How it works

Here's the common start process of rod:

  1. Try to connect to a Devtools endpoint (WebSocket), if not found try to launch a local browser, if still not found try to download one, then connect again. The lib to handle it is launcher.

  2. Use the JSON-RPC to talk to the Devtools endpoint to control the browser. The lib handles it is cdp.

  3. Use the type definitions of the JSON-RPC to perform high-level actions. The lib handles it is proto.

Object model:

object model

Join us

Your help is more than welcome! Even just open an issue to ask a question may greatly help others.

Please read How To Ask Questions The Smart Way before you ask questions.

We use Github Projects to manage tasks, you can see the priority and progress of the issues here.

If you want to contribute code for this project please read the Contributor Guide.