Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.
We also have a TypeScript implementation of the Crawlee, which you can explore and utilize for your projects. Visit our GitHub repository for more information Crawlee on GitHub.
Crawlee is available as the crawlee PyPI package.
pip install crawleeAdditional, optional dependencies unlocking more features are shipped as package extras.
If you plan to use BeautifulSoupCrawler, install crawlee with beautifulsoup extra:
pip install 'crawlee[beautifulsoup]'If you plan to use PlaywrightCrawler, install crawlee with the playwright extra:
pip install 'crawlee[playwright]'Then, install the Playwright dependencies:
playwright installYou can install multiple extras at once by using a comma as a separator:
pip install 'crawlee[beautifulsoup,playwright]'- Unified interface for HTTP and headless browser crawling.
- Persistent queue for URLs to crawl (breadth & depth-first).
- Pluggable storage of both tabular data and files.
- Automatic scaling with available system resources.
- Integrated proxy rotation and session management.
- Configurable request routing - directing URLs to appropriate handlers.
- Robust error handling.
- Automatic retries when getting blocked.
- Written in Python with type hints, which means better DX and fewer bugs.
Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
Crawlee offers a framework for parallel web crawling through a variety of crawler classes, each designed to meet different crawling needs.
HttpCrawler provides a framework
for the parallel crawling of web pages using plain HTTP requests. The URLs to crawl are fed from a request provider.
It enables the recursive crawling of websites. The parsing of obtained HTML is the user's responsibility.
Since HttpCrawler uses raw HTTP requests to download web pages, it is very fast and efficient on data
bandwidth. However, if the target website requires JavaScript to display the content, you might need to use
some browser crawler instead, e.g. PlaywrightCrawler, because it loads the pages using a full-featured headless Chrome browser.
HttpCrawler downloads each URL using a plain HTTP request, obtain the response and then invokes the
user-provided request handler to extract page data.
The source URLs are represented using the
Request objects that are fed from
RequestList
or RequestQueue
instances provided by the request provider option.
The crawler finishes when there are no more Request objects to crawl.
If you want to parse data using BeautifulSoup see
the BeautifulSoupCrawler section.
Example usage:
import asyncio
from crawlee.http_crawler import HttpCrawler, HttpCrawlingContext
async def main() -> None:
# Create a HttpCrawler instance and provide a starting requests
crawler = HttpCrawler()
# Define a handler for processing requests
@crawler.router.default_handler
async def request_handler(context: HttpCrawlingContext) -> None:
# Crawler will provide a HttpCrawlingContext instance,
# from which you can access the request and response data
data = {
'url': context.request.url,
'status_code': context.http_response.status_code,
'headers': dict(context.http_response.headers),
'response': context.http_response.read().decode()[:1000],
}
# Extract the record and push it to the dataset
await context.push_data(data)
# Add first URL to the queue and start the crawl
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())For further explanation of storages (dataset, request queue) see the storages section.
BeautifulSoupCrawler extends
the HttpCrawler. It provides the same features and on top of that, it uses
BeautifulSoup HTML parser.
Same as for HttpCrawler, since BeautifulSoupCrawler uses raw HTTP requests to download web pages,
it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display
the content, you might need to use PlaywrightCrawler instead, because it loads the pages using
a full-featured headless browser (Chrome, Firefox or others).
BeautifulSoupCrawler downloads each URL using a plain HTTP request, parses the HTML content using BeautifulSoup
and then invokes the user-provided request handler to extract page data using an interface to the
parsed HTML DOM.
Example usage:
import asyncio
from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main() -> None:
# Create a BeautifulSoupCrawler instance
crawler = BeautifulSoupCrawler()
# Define a handler for processing requests
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
# Crawler will provide a BeautifulSoupCrawlingContext instance,
# from which you can access the request and response data
data = {
'title': context.soup.title.text,
'url': context.request.url,
}
# Extract the record and push it to the dataset
await context.push_data(data)
# Add first URL to the queue and start the crawl
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())BeautifulSoupCrawler also provides a helper for enqueuing links in the currently crawling website.
See the following example with the updated request handler:
from crawlee.enqueue_strategy import EnqueueStrategy
# ...
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
# Use enqueue links helper to enqueue all links from the page with the same domain
await context.enqueue_links(strategy=EnqueueStrategy.SAME_DOMAIN)
data = {
'title': context.soup.title.text,
'url': context.request.url,
}
await context.push_data(data)PlaywrightCrawler extends
the BasicCrawler. It provides the same features and on top of that, it uses
Playwright browser automation tool.
This crawler provides a straightforward framework for parallel web page crawling using headless versions of Chromium,
Firefox, and Webkit browsers through Playwright. URLs to be crawled are supplied by a request provider, which can be
either a RequestList containing a static list of URLs or a dynamic RequestQueue.
Using a headless browser to download web pages and extract data, PlaywrightCrawler is ideal for crawling
websites that require JavaScript execution. For websites that do not require JavaScript, consider using
the BeautifulSoupCrawler, which utilizes raw HTTP requests and will be much faster.
Example usage:
import asyncio
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
# Create a crawler instance
crawler = PlaywrightCrawler(
# headless=False,
# browser_type='firefox',
)
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
data = {
'request_url': context.request.url,
'page_url': context.page.url,
'page_title': await context.page.title(),
'page_content': (await context.page.content())[:10000],
}
await context.push_data(data)
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())Example usage with custom browser pool:
import asyncio
from crawlee.browsers import BrowserPool, PlaywrightBrowserPlugin
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
# Create a browser pool with a Playwright browser plugin
browser_pool = BrowserPool(
plugins=[
PlaywrightBrowserPlugin(
browser_type='firefox',
browser_options={'headless': False},
page_options={'viewport': {'width': 1920, 'height': 1080}},
)
]
)
# Create a crawler instance and provide a browser pool
crawler = PlaywrightCrawler(browser_pool=browser_pool)
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
data = {
'request_url': context.request.url,
'page_url': context.page.url,
'page_title': await context.page.title(),
'page_content': (await context.page.content())[:10000],
}
await context.push_data(data)
await crawler.run(['https://apify.com', 'https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())Crawlee introduces several result storage types that are useful for specific tasks. The storing of underlying data is realized by the storage client. Currently, only a memory storage client is implemented. Using this, the data are stored either in the memory or persisted on the disk.
By default, the data are stored in the directory specified by the CRAWLEE_STORAGE_DIR environment variable.
With default .storage/.
A Dataset is a type
of storage mainly suitable for storing tabular data.
Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. The dataset can be imagined as a table, where each object is a row and its attributes are columns. The dataset is an append-only storage - we can only add new records to it, but we cannot modify or remove existing records.
Each Crawlee project run is associated with a default dataset. Typically, it is used to store crawling results specific to the crawler run. Its usage is optional.
The data are persisted as follows:
{CRAWLEE_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json
The following code demonstrates the basic operations of the dataset:
import asyncio
from crawlee.storages import Dataset
async def main() -> None:
# Open a default dataset
dataset = await Dataset.open()
# Push a single record
await dataset.push_data({'key1': 'value1'})
# Get records from the dataset
data = await dataset.get_data()
print(f'Dataset data: {data.items}') # Dataset data: [{'key1': 'value1'}]
# Open a named dataset
dataset_named = await Dataset.open(name='some-name')
# Push multiple records
await dataset_named.push_data([{'key2': 'value2'}, {'key3': 'value3'}])
if __name__ == '__main__':
asyncio.run(main())The KeyValueStore
is used for saving and reading data records or files. Each data record is represented by a unique
key and associated with a MIME content type. Key-value stores are ideal for saving screenshots of web pages, and PDFs
or to persist the state of crawlers.
Each Crawlee project run is associated with a default key-value store. By convention, the project input and output are stored in the default key-value store under the INPUT and OUTPUT keys respectively. Typically, both input
and output are JSON files, although they could be any other format.
The data are persisted as follows:
{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
The following code demonstrates the basic operations of key-value stores:
import asyncio
from crawlee.storages import KeyValueStore
async def main() -> None:
kvs = await KeyValueStore.open() # Open a default key-value store
# Write the OUTPUT to the default key-value store
await kvs.set_value('OUTPUT', {'my_result': 123})
# Read the OUTPUT from the default key-value store
value = await kvs.get_value('OUTPUT')
print(f'Value of OUTPUT: {value}') # Value of OUTPUT: {'my_result': 123}
# Open a named key-value store
kvs_named = await KeyValueStore.open(name='some-name')
# Write a record to the named key-value store
await kvs_named.set_value('some-key', {'foo': 'bar'})
# Delete a record from the named key-value store
await kvs_named.set_value('some-key', None)
if __name__ == '__main__':
asyncio.run(main())The RequestQueue
is a storage of URLs (requests) to crawl. The queue is used for the deep crawling of websites,
where we start with several URLs and then recursively follow links to other pages. The data structure supports both
breadth-first and depth-first crawling orders.
Each Crawlee project run is associated with a default request queue. Typically, it is used to store URLs to crawl in the specific crawler run. Its usage is optional.
The data are persisted as follows:
{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.json
The following code demonstrates the basic usage of the request queue:
import asyncio
from crawlee.storages import RequestQueue
async def main() -> None:
# Open a default request queue
rq = await RequestQueue.open()
# Add a single request
await rq.add_request('https://crawlee.dev')
# Open a named request queue
rq_named = await RequestQueue.open(name='some-name')
# Add multiple requests
await rq_named.add_requests_batched(['https://apify.com', 'https://example.com'])
# Fetch the next request
request = await rq_named.fetch_next_request()
print(f'Next request: {request.url}') # Next request: https://apify.com
if __name__ == '__main__':
asyncio.run(main())For an example of usage of the request queue with a crawler see the BeautifulSoupCrawler example.
SessionPool is a class that allows us to handle the rotation of proxy IP addresses along with cookies and other custom settings in Crawlee.
The main benefit of using a session pool is that we can filter out blocked or non-working proxies, so our actor does not retry requests over known blocked/non-working proxies. Another benefit of using the session pool is that we can store information tied tightly to an IP address, such as cookies, auth tokens, and particular headers. Having our cookies and other identifiers used only with a specific IP will reduce the chance of being blocked. The last but not least benefit is the even rotation of IP addresses - the session pool picks the session randomly, which should prevent burning out a small pool of available IPs.
To use a default session pool with automatic session rotation use the use_session_pool option for the crawler.
from crawlee.http_crawler import HttpCrawler
crawler = HttpCrawler(use_session_pool=True)If you want to configure your own session pool, instantiate it and provide it directly to the crawler.
import asyncio
from datetime import timedelta
from crawlee.http_crawler import HttpCrawler
from crawlee.sessions import Session, SessionPool
async def main() -> None:
# Use dict as args for new sessions
session_pool_v1 = SessionPool(
max_pool_size=10,
create_session_settings = {'max_age': timedelta(minutes=10)},
)
# Use lambda creation function for new sessions
session_pool_v2 = SessionPool(
max_pool_size=10,
create_session_function=lambda _: Session(max_age=timedelta(minutes=10)),
)
crawler = HttpCrawler(session_pool=session_pool_v1, use_session_pool=True)
if __name__ == '__main__':
asyncio.run(main())Crawlee is open-source and runs anywhere, but since it's developed by Apify, it's easy to set up on the Apify platform and run in the cloud. Visit the Apify SDK website to learn more about deploying Crawlee to the Apify platform.
If you find any bug or issue with Crawlee, please submit an issue on GitHub. For questions, you can ask on Stack Overflow, in GitHub Discussions or you can join our Discord server.
Your code contributions are welcome, and you'll be praised for eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see CONTRIBUTING.md.
This project is licensed under the Apache License 2.0 - see the LICENSE.md file for details.