A crawler originally designed for detecting weibo deletion. One can use it to accurately track a few weibo users, i.e., detecting weibo creation/deletion, outputting all weibos sent by a user to a csv file, and capturing screenshots of weibos as well as hot comments. Note that this crawler is "accurate", which means it is not intended to be used to crawl huge amount of weibos aimlessly.
- Crawl weibos of some users
- Check if the user post some weibos
- Check if the user delete some weibos
- Capture screenshots of weibos
- Capture screenshots of the hot comments of weibos
- Export weibos as csv format
- Export deleted weibos as csv format
- Crawl users' comments
- Instagram support
- Twitter support
- Echo support
- Install requirements
- Python 2.7.x
- pip
- git
- PhantomJS
- virtualenv (Recommended)
- Clone the git:
git clone https://github.com/wdwind/accurate_weibo_crawler
- Go into the new directory:
cd accurate_weibo_crawler
- (Optional) Create a virtual environment:
virtualenv .env
- (Optional) Activate the virtual environment:
.env\Scripts\activate
- Install dependencies:
pip install -r requirements.txt
Copy file config.json.example
and rename it as config.json
. All configuration options are in config.json
.
mode
: The mode of the crawler.
normal
: Crawl weibo once; capture screenshots once.weibo
: Only crawl weibo once.screenshot
: Only capture screenshots once.loop
: Loop the normal mode every 5 minutes.export
: Export all crawled weibo to csv file(s).
-
db
: Database. Let the default value here except if you want to use multiple databases. -
phantomjs
: The path to the executable PhantomJS binary, e.g.,C:\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe
. -
timeout
: Timeout when capturing the screenshots (in milliseconds). This value should be set based on your Internet connection speed. -
access_token
andexpires_in
. Go to the following link:Finish the authorization. And then you will be redirected to a blank page where you can get the
access_token
andexpires_in
from the url:http:// oauth.weico.cc/#access_token=【This is your
access_token
】&remind_in=xxx&expires_in=【This is yourexpires_in
】&refresh_token=xxx&uid=xxx&scope=xxx. -
task
: The specific accounts and tasks you would like the crawler to perform.uid
: Id of the user to be crawled. This id is contained in the url of a user's homepage.name
: User's name. You can input anything here.screenshots_path
: The path to store the screenshots.temp_screenshots_path
: The path to store the temporary screenshots. Because the quality of the screenshot highly depends on the network connectivity, sometimes the screenshots are just nothing. This folder let you check the quality of screenshots manually, and move the good ones intoscreenshots_path
.screenshots
: Set whether to capture the screenshots of weibo.screenshots_hot_comments
: Set whether to capture the screenshots of the hot comments. Currently Weibo disables hot comments in m.weibo.cn, so this option should be kept asfalse
.
python crawler.py
Since Weibo's (public or private) APIs are extremely unstable, there is no guarantee that the output of the program is correct. Use it at your own risk.