![Screenshot 2024-04-20 at 23 11 16](https://private-user-images.githubusercontent.com/10400064/324205393-ab00e048-a9ff-43b6-81d5-2e58090e2e65.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYwMzk3NTcsIm5iZiI6MTcyNjAzOTQ1NywicGF0aCI6Ii8xMDQwMDA2NC8zMjQyMDUzOTMtYWIwMGUwNDgtYTlmZi00M2I2LTgxZDUtMmU1ODA5MGUyZTY1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTExVDA3MjQxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg5NGI4ZWUxYTNjZjVlZDY4ZmMxZmI5ODZkMDljYTdmY2M3MGQ1ZWIyYmJkZjM4ZGZmYTEyMjNjYzQzM2RjOWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.55DccjyvA3f4w6DOMs_yJIA1t-DrsiQBlWh-MPxiqLY)
LLM Scraper is a TypeScript library that allows you to convert any webpages into structured data using LLMs.
Tip
Under the hood, it uses function calling to convert pages to structured data. You can find more about this approach here
- Supports OpenAI, Groq chat models
- Schemas defined with Zod
- Full type-safety with TypeScript
- Based on Playwright framework
- Streaming when crawling multiple pages
- Supports 4 input modes:
html
for loading raw HTMLmarkdown
for loading markdowntext
for loading extracted text (using Readability.js)image
for loading a screenshot (multi-modal only)
Make sure to give it a star!
![Screenshot 2024-04-20 at 22 13 32](https://private-user-images.githubusercontent.com/10400064/324202813-11e2a79f-a835-48c4-9f85-5c104ca7bb49.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYwMzk3NTcsIm5iZiI6MTcyNjAzOTQ1NywicGF0aCI6Ii8xMDQwMDA2NC8zMjQyMDI4MTMtMTFlMmE3OWYtYTgzNS00OGM0LTlmODUtNWMxMDRjYTdiYjQ5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTExVDA3MjQxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdkYTkzNGVmZmVhYjkxMTAyZGExZWM2MjlhNWIwMjFmYzEzMzNhZWFhZWUxODg1ODQ3YThkYTFhNTc3NDkyOGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.wJMCljy0dAW13Equge74NYFTyd1f0vyYdnFDBheUAbY)
-
Install the required dependencies from npm:
npm i zod playwright llm-scraper
-
Get an OpenAI API key and set it in your environment variables:
export OPENAI_API_KEY=***
-
Create a new browser instance and attach LLMScraper to it:
import { chromium } from 'playwright' import LLMScraper from 'llm-scraper' const browser = await chromium.launch() const scraper = new LLMScraper(browser, 'gpt-4-turbo', { /* model config */ })
In this example, we're extracting top stories from HackerNews:
import { chromium } from 'playwright'
import { z } from 'zod'
import LLMScraper from 'llm-scraper'
// Create a new browser instance
const browser = await chromium.launch()
// Initialize the LLMScraper instance
const scraper = new LLMScraper(browser, 'gpt-4-turbo')
// Define schema to extract contents into
const schema = z.object({
top: z
.array(
z.object({
title: z.string(),
points: z.number(),
by: z.string(),
commentsURL: z.string(),
})
)
.length(5)
.describe('Top 5 stories on Hacker News'),
})
// URLs to scrape
const urls = ['https://news.ycombinator.com']
// Run the scraper
const pages = await scraper.run(urls, {
schema,
mode: 'html',
closeOnFinish: true,
})
// Stream the result from LLM
for await (const page of pages) {
console.log(page.data)
}
As an open-source project, we welcome contributions from the community. If you are experiencing any bugs or want to add some improvements, please feel free to open an issue or pull request.