Calculate daily epu index
Closed this issue · 7 comments
Start a job after running the scrapers that will calculate and persist the epu index of yesterday.
@niconoe is it ok if I forward this to you to make it a Django command? I think we will schedule the scraper commands with cron, and when all of them are finished, this job should run.
Yep!
For more flexibility and being able to deal with cron misconfiguration issues, shouldn't this command allow:
- either to run for a specific day given as a parameter
- either for the "yesterday" special option (used by default)
- either reprocess everything?
Indeed. However, "everything" will be difficult. Since for days before 2014, we don't know the number of journals scraped. (In fact, maybe we need a place to store this for new data too?)
@niconoe apparently the cutoff for the epu index is not 0 but -0.15.
So the EPU index is the number of articles with a epu score higher than -0.15 divided by the number of journals scraped.
So again, maybe we need a place where we can store the number of journals scraped. Some place where every scraper can write "I succeeded for this day". Maybe a table "journals scraped" with two columns "date" and "spider/journal name"?
See #64
It's now implemented, use as:
$ python manage.py calculate_daily_epu 2015-08-17
It tells what it does on stdout and store its result in EpuIndexScore. Please review and test!
Tested. Works perfectly.