Stats cron job sometimes times out
Closed this issue · 2 comments
The last few days, the stats cron job has failed with Updating stats failed: failed reading Song.Tags: API error 5 (datastore_v3: TIMEOUT): The datastore operation timed out, or the data was temporarily unavailable.
It looks like it took a bit over a minute in the preceding days, and often 30-40 seconds before then. I'm not sure if the timeout I'm hitting is an internal AppEngine one or datastore-specific. If it's just datastore, I can probably split this into multiple queries using cursors.
I (accidentally?) wasn't parallelizing the projection queries on Song properties. After doing that, the job completed successfully, but it's still quite slow:
16:49:26.898 /stats?update=1
16:49:28.702 Computing Song.AlbumId stats took 1040 ms
16:49:38.159 Computing Song.Length stats took 10497 ms
16:49:40.448 Computing Song.Rating stats took 12786 ms
16:50:28.134 Computing Song.Tags stats took 60472 ms
16:51:16.185 Computing Play stats took 48050 ms
http://googleappengine.blogspot.com/2010/12/happy-holidays-from-app-engine-team-140.html says:
No more 30-second limit for background work - With this release, we’ve significantly raised this limit for offline requests from Task Queue and Cron: you can now run for up to 10 minutes without interruption.
So presumably the limit I was hitting is datastore-specific. I guess I should look into using cursors.
https://stackoverflow.com/q/31332757 suggests that datastore queries have a one-minute limit.