DOMSpider Spider base Scrapy for crawling web 2.0 page usage: install PySide, scrapy, redis run redis-server at default port cd DOMSpider scrapy crawl urls change urls_spider.py to crawl your site.