Electric Capital Crypto Ecosystems Crawler

Overview

This crawler script, designed for the Electric Capital Crypto Ecosystems repository, automates the process of updating the list of LENS-related projects on GitHub. It identifies new projects using specific Lens Protocol dependencies, compares them with the existing list in lens-protocol.toml, and creates a pull request in the Electric Capital repository with new Lens protocol projects.

Features

Automated GitHub Crawling: Searches for specific LENS-related keywords in package.json, package-lock.json, yarn.lock, and .sol files.
Intelligent Filtering: Adds only new projects not already listed in lens-protocol.toml.
Automated Pull Requests: Creates a pull request with updates on a forked Electric Capital repository.
Rate Limit Handling: Manages GitHub API rate limits.

Requirements

Python 3.x
requirements.txt
A GitHub personal access token

Setup

Environment Variables: Set GITHUB_TOKEN and GITHUB_USERNAME in a .env file. (You can copy .env.example to .env, and edit it)
Dependencies: Install required Python libraries with pip install -r requirements.txt command.

Usage

Run python3 main.py on the root folder. The script then,

Initializes and loads environment variables.
Searches GitHub for repositories with specified LENS-related criteria.
Compares and filters found repositories against lens-protocol.toml.
Synchronizes the forked repository with the upstream repository.
Creates a pull request with new entries in the forked repository.

Output

Logs of the process.
A link to pull request in the forked repository with lens-protocol.toml updates.

Error Handling

Handles GitHub API rate limits and HTTP request failures.

Contribution

Contribute by forking the repository, making changes, and submitting a pull request.

Note: Comply with GitHub's API policies and rate limits when using this script.