Scrapy project
An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
Pinned Repositories
cssselect
CSS Selectors for Python
dirbot
Scrapy project to scrape public web directories (educational) [DEPRECATED]
parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
quotesbot
This is a sample Scrapy project for educational purposes
scrapely
A pure-python HTML screen-scraping library
scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
scrapy.org
The scrapy.org website
scrapyd
A service daemon to run Scrapy spiders
scrapyd-client
Command line client for Scrapyd server
w3lib
Python library of web-related functions
Scrapy project's Repositories
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
scrapy/scrapyd
A service daemon to run Scrapy spiders
scrapy/scrapely
A pure-python HTML screen-scraping library
scrapy/dirbot
Scrapy project to scrape public web directories (educational) [DEPRECATED]
scrapy/quotesbot
This is a sample Scrapy project for educational purposes
scrapy/parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
scrapy/scrapyd-client
Command line client for Scrapyd server
scrapy/w3lib
Python library of web-related functions
scrapy/cssselect
CSS Selectors for Python
scrapy/loginform
Fill HTML login forms automatically
scrapy/queuelib
Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
scrapy/slybot
scrapy/itemadapter
Common interface for data container classes
scrapy/scrapy.org
The scrapy.org website
scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
scrapy/itemloaders
Library to populate items using XPath and CSS with a convenient API
scrapy/booksbot
A crawler for http://books.toscrape.com
scrapy/scrapy-bench
A CLI for benchmarking Scrapy.
scrapy/scurl
Performance-focused replacement for Python urllib
scrapy/pypydispatcher
A fork of http://pydispatcher.sourceforge.net/ with PyPy support
scrapy/xtractmime
https://mimesniff.spec.whatwg.org/ implementation for Python
scrapy/base-chromium
base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
scrapy/scrapy-itemloader
[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
scrapy/gsoc2014-integration-tests
GSoC2014 - Scrapy Integration tests project
scrapy/scrapy-bench-speedcenter
Codespeed for scrapy-bench
scrapy/url-chromium
url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url