scrapinghub/spidermon

Add handling for datetime aware start time in PeriodicExecutionTimeMonitor

Closed this issue · 1 comments

In the current version of scrapy, the code below breaks due to spider start time being timezone aware
https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/monitors/monitors.py#L497

now = datetime.datetime.utcnow()
start_time = self.data.stats.get("start_time")
duration = now - start_time

It currently throws the following error:

TypeError: can't subtract offset-native and offset-aware datetimes

Backwards compatible Fix: add try/except to add utc timezone to now timestamp when error thrown.

#438 took care of this issue.