everypolitician/scraped

Check that responses have correct status

Opened this issue · 0 comments

Problem

If a response fails and returns a non-200 error we currently don't do anything about it and pass the response untouched into the scraper class. It would be useful if Scraped did some kind of error checking on the status code.

Proposed solution

By default, we should check for a 200 response and then either print a warning or raise an exception if there is a non-200 response. This should be configurable so that a class can change the expected status, perhaps with a class level expected_status 201 type macro.