domingosmiguel/webcrawler

Web Scraper

Overview

This C# web scraper project uses Selenium and HtmlAgilityPack to collect proxy server data from proxyservers.pro and saves it in various formats (HTML, JSON, and Database).

Prerequisites

• .NET SDK installed
• ChromeDriver installed and added to your system's PATH.

Getting Started

1. Clone this repository:

   git clone https://github.com/your-username/web-scraper.git

2. Navigate to the project directory:

   cd web-scraper

3. Download dependencies:

   dotnet restore

4. Apply database migrations:

   dotnet ef database update

5. Build the project:

   dotnet build

6. Run the project:

   dotnet run

Configuration

• The base URL for scraping is set in the baseUrl variable in the Program.cs file.
• The project uses SQLite as the default database. You can change the database configuration in the DataContext.cs file.

License

This project is licensed under the MIT License - see the LICENSE file for details.