/scrapy-googledir-mysql

scrapy's googledir example with mysql store pipeline using twisted's adbapi

Primary LanguagePython

1. create database
    $ mysql -uroot
    mysql> create database googledir;
2. create table
    $ mysql -uroot googledir < db/mysql.sql
3. start crawling
    $ ./scrapy-ctl.py crawl directory.google.com

DB credentials are hardcoded in `googledir/pipelines.py`