codeigniter based price crawler using Xpath to crawl ecommerce data.
The ideas for user is:
- Entry new website
- Give example url containing product detail consist of name of product and price.
- Give xpath location for price and name
- Give starter url to crawl.
- (optional) give regex formula to match product url (to speedup crawling, only check for product page only based on regex match)
- Get data grow for link and product price summary.
Behind the scene: We need to put 2 url in cronjob. 1st script is collecting the URL 2nd script is trying to extract data from those URL.
In the end of the day you got bunch of price data.
Prequisite:
- PHP 5.2 and above
- Apache or nginx
- MySQL
How to install:
- Dump database.sql
- Edit application/config/config.php change the base url to your own.
- Modify database credential in application/config/database.php
- Put linkminer script into cronjob : http://your-host.com/crawler/crawler/linkminer/ (ideally every 5 minutes)
- Put contentminer script into cronjob : http://your-host.com/crawler/crawler/contentminer/ (ideally every 5 minute)
That's for now.