Generate consolidated text files from websites for LLM training and inference. Powered by @firecrawl_dev for web crawling and GPT-4-mini for text processing.
- Crawls websites and combines content into a single text file
- Generates both standard (
llms.txt) and full (llms-full.txt) versions - Web interface and API access available
- No API key required for basic usage
Visit llmstxt.firecrawl.dev to generate files through the browser.
GET https://llmstxt.firecrawl.dev/[YOUR_URL_HERE]
Note: Processing may take several minutes due to crawling and LLM operations.
Create a .env file with the following variables:
FIRECRAWL_API_KEY=
SUPABASE_URL=
SUPABASE_KEY=
OPENAI_API_KEY=
npm install
npm run dev