webcrawlers

There are 8 repositories under webcrawlers topic.

  • amplify-spiders-v1

    ExcitingTheory/amplify-spiders-v1

    A tool for search engine based competitive analysis. AWS Amplify project with many web crawlers for different search engines, and a simple web frontend for displaying historical results in Next.js

    Language:TypeScript7000

  • AnonCatalyst/WebDiver

    WebDiver is a versatile Python script for crawling websites, extracting internal and external links, titles, and descriptions. It's useful for tasks such as web analysis, OSINT (Open Source Intelligence) gathering, and competitive analysis.

    Language:Python6101

  • ozakboy/Taiwan-news-crawlers

    .net-based Crawlers for news of Taiwan (.net 台灣新聞爬蟲，數據物件化，方便使用)

    Language:C#20

  • alimogh/spideyweb

    Simple WebCrawler

    Language:Python100

  • mr-rjh3/webcrawlers

    Various webcrawler programs made to gain a deeper understandings of webcrawling concepts.

    Language:Python120

  • Th3-C0der/Web-Crawler

    A simple WebCrawler for exploring and downloading content from web pages within a given domain/url.

    Language:HTML1100

  • Modyev/WebsiteCrawler

    Web Crawler written in C# that parses all urls from a specific page then recursively visits them while parsing all links available on that webpage

    Language:C#

  • slemarchand/no-robots

    🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.

    Language:Java20