/gospider

🚀Gospider is a powerful Golang web crawler that includes all the necessary libraries for transitioning from Python to Golang. It provides a fast and seamless transition for Python web crawlers to Golang.

Primary LanguageGoGNU Lesser General Public License v3.0LGPL-3.0

Introduction

Gospider is a powerful Golang web crawler that includes all the necessary libraries for transitioning from Python to Golang. It provides a fast and seamless transition for Python web crawlers to Golang.

Module documentation can be found at the following link!!!

Module documentation can be found at the following link!!!

Module documentation can be found at the following link!!!

  1. Request Library: JA3 and HTTP/2 fingerprinting. Websocket, SSE, HTTP, and HTTPS protocols.
  2. Concurrency Library: Self-implemented high-performance concurrency library.
  3. Execute JavaScript and Python: Invoke methods in JavaScript and Python through pipes.

Dependencies

go1.21 (Do not use a version lower than this)

Installation (Do not fetch the package from GitHub, choose either Gitee or GitHub for the go package path. Fetching from GitHub will cause path issues.)

go get -u gitee.com/baixudong/gospider

For easy management, please submit bugs on GitHub

https://github.com/baixudong007/gospider

Test Cases

Recommended Libraries

Library Name Reason for Recommendation
curl_cffi The best library for modifying JA3 fingerprints in Python.
chromedp The best library for browser manipulation in Golang.