ai-crawler
There are 13 repositories under ai-crawler topic.
firecrawl/firecrawl
The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data 🔥
oxylabs/oxylabs-ai-studio-py
Oxylabs AI Studio python SDK
oxylabs/ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
kr3t3n/smolagents-video-script-generator
A sophisticated system that uses multiple AI agents to research, create, and polish video scripts for social media platforms. The system employs specialized agents for research, script writing, polishing, and evaluation to ensure high-quality, engaging content.
lennyerik/crawl4ai-proxy
A simple proxy server to integrate crawl4ai with OpenWebUI
spider-rs/spider-cloud-live-code-viewer
Visual web crawler
JerryZhi/AI-Crawler-Detector
Tool for Fast Detection of Website/Server AI Crawler Blocking Policies（Not robots.txt）
oxylabs/oxylabs-ai-studio-js
Oxylabs AI Studio JS SDK
breadrock1/news-rss
There is simple project to scrape and collect news using rss and llm API based on rust.
kr3t3n/documentation-crawler
A powerful tool that crawls documentation websites and generates a clean, well-formatted markdown document. Built with FastAPI and support for multiple LLM providers (DeepSeek and Groq).
sonhm3029/Auto-Crawl-Any-Page-with-Large-Language-Model
Auto Crawl Any Page with Large Language Model
RMNCLDYO/firecrawl-toolkit
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.
MubeenAk47/ai-ready-website
🚀 Analyze your website's AI readiness and optimize for performance with real-time scoring, recommendations, and detailed metrics.