Datafable/epu-index

Alert needed when scraper fails?

Closed this issue · 7 comments

bartaelterman commented

Should the user be alerted (via email for instance) if a scraper fails?

bartaelterman commented

Here is what we will do:

A separate check script will run after all scrapers have run. If one spider returned 0 articles for 7 consecutive days, an email alert is sent.

bartaelterman commented

@niconoe I think another django command would be best for this, so I'll assign you. Nevetheless, I can help with the code to email (example here)

niconoe commented

This is implemented, it use the Django machinery for sending mails, so at the deployment this will have to be configured/tested the Django way.

bartaelterman commented

Excellent. I've create an account for the client. Will test it.

bartaelterman commented

Shouldn't it get a password somewhere?

niconoe commented

In production, I guess you would use SMTP and then have to configure the following settings: https://docs.djangoproject.com/en/1.8/topics/email/#smtp-backend (probably at least the first four settings).

So far, I tested it locally using the console backend.

bartaelterman commented

👍

Indeed the first four settings need to be set (this can help). Also EMAIL_USE_SSL needs to be set to True. Besides setting these settings, no other action is needed.

Tested it, and emails are indeed sent as expected!