Enviroment setup
1. Python 2.7 (We recommend to set it up with anaconda)
2. Install the packages specified in requirements.txt
pip install -r requirements.txt
3. Download Phantomjs here for Windows, here for MacOS and here for Linux-64 bit( Other references), install it and add the path to 'phantomjsPath' of config.yml.
Populate the config.yml file :
-
"UserName" is the username for coursera account;
-
"Password" is the password for coursera account;
-
"UserId" is the ID for every account.
First, you have to login using your coursera account on the website. Then press F12, select "Network" and choose XHR. You can see an API link "https://www.coursera.org/api/openCourseMemberships.v1/?q=findByUser&userId=XXX", your userid is "XXX". -
"filePath" is the path to save the data you crawled. Make a 'data' folder in the same directory and add the relative path to it here. eg. './data/'
When you login, you will see "My Courses", including "Last Active" and "Inactive". -
"activeCoursePageNum" is the maximum pages of your "Last Active" courses you want to crawl.
-
"inactiveCoursePageNum" is the maximum pages of your "Inactive" courses you want to crawl.
-
"phantomjsPath" is the executable file path of phantomjs.
-
"dbPath" is the location of the sqlite3 database file on disk. For fist run - make a blank database.
-
"debug" : set it to one to see debug messages in the terminal on running the code, otherwise set it to 0.
-
"cookie" : First sign in to your coursera account on any web browser. Add the cookie after logging into the coursera account using the browser.Instructions to get the cookie variable - http://www.whatarecookies.com/view.asp
python incremental_crawler.py
The crawled data will be saved in folders. Every course has a folder named as courseName_courseID_crawlTime(%Y_%m_%d)