crawler-engine
There are 51 repositories under crawler-engine topic.
6677-ai/tap4-ai-crawler
The crawler opened source by tap4.ai
nuhmanpk/WebScrapper
Simple and powerfull all in one Telegram Bot to scrap / crawl webpages using Requests, html5lib and Beautifulsoup
RevoltSecurities/SpideyX
SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.
namhong1412/browser-clone-web
Use browser to re-copy a web page
bkeepers/spiderman
your friendly neighborhood web crawler
fooock/robots.txt
:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
web-extractors/arachnid-seo-js
Web crawler for extracting internal site links info for SEO auditing & optimization purposes
Sobak/scrawler
Declarative, scriptable web robot (crawler) and scrapper
wefindx/metadrive
Generic Interfaces to Addressable Objects
wetrycode/tegenaria
Tegenaria is a crawler framework based on golang
crawlbase/crawlbase-ruby
Fast Crawlbase API crawling library
BaseMax/NetPHP
Useful functions for connecting to the network in the PHP based applications.
lichang98/visualize_spider
基于Spring Boot、Scrapy 的可视化爬虫配置与管理
ShiqinHuo/wuhan_house_price_crawler
武汉东湖高新片区光谷&软件园二手房房价爬虫。data source: 房天下
spekulatius/spatie-crawler-cached-queue-example
Example to demonstrate the usage of cached queues across multiple requests.
supernebula/shark
Shark (Plunder)可配置、插件化的爬虫引擎，二次开发框架。Configurable, pluginable crawler engine, secondary development framework.
hseghetti/simple-crawler
Simple crawler using apache nutch and elasticsearch
MCStreetguy/Crawler
An advanced web-crawler written in PHP.
andrrff/BugSearch
BugSearch é um motor de pesquisa de páginas indexadas pelo crawler BugSearch.Crawler. O projeto é dividido em duas partes: o lado do Bot (Bot side) e o lado do Cliente (Client side).
Colaplusice/zhihu
数据挖掘实验，抓取用户信息并且进行聚类等处理
its-my-data/android-crawler-engine
An Android app crawling framework, making automatic crawling mobile apps super easy! (if possible, iOS will be supported after Android version)
KonghaYao/jspider
This is a JavaScript toolkit for browser crawler testing.
plugnsearch/plugnsearch
The only real pluggable crawler / spider / webcrawler to search the web for stuff you need to know.
takadev15/onecrawl-rs
Blazingly Fast, High Performant, Scalable Web Crawler Engine 💨
johnvanderton/flysh
HTML type document parser based on jQuery and JSDOM
Keerthivasan13/Targeted_Advertising_Google_AdSense
Hybrid E-Marketing using Web Page Mining for Website Monetization
kingzbauer/scraperlang
A DSL aimed at making writing web scrapers/crawlers a breeze
rihenperry/whirlpool-urlfrontier
mercator scheme/rate-limiting/scheduling part of whirlpool project; handles crawler priority and politeness
robincloud/robinbot
robin micro web crawling engine with nodejs
rrmerugu/trawler
A data gathering/trawling framework to search and get information from web sources like bing
runjia1987/crawler-engine
crawler-engine with HTTP, proxy, JS-Java Interoperability, MQ task consumption, dynamic crawler scripts execution. support deployment in distribution style.
eyazdpour/DirectoryCrawler
Simple crawler for a directory (on Windows) which return all possible information about whatever is in that given directory
MaximeGuinard/Gtool-projects-crawler-seo
🤖 A Google extension that facilitates project management with various tools
paganini2008/greenfinger
A high-performance distributed web crawling framework based on SpringBoot framework. It provides rich APIs to customize business and easily embedded your system.
setulparmar/Landslide-Detection-and-Prediction
This project named "Landslide Detection and Prediction" was done during my summer internship under Visiting Associate Prof. Gagan Raj Gupta at IIT - Bhilai.
ShubhamThakurela/global-social-media-ms
Functionality to Extract Social data.