openspending/spendb

ETL can't perform clean up due to S3 bad request

Opened this issue · 0 comments

After finally finding somewhere for the bucket where data can be uploaded, the app doesn't appear to be able to get the data back out. The user has S3 full access. I suppose there is a small chance this is the reason for #35...

Debug:
[2015-12-07 23:11:32,967: WARNING/Worker-2] Failed: Clean up source data
[2015-12-07 23:11:33,693: ERROR/MainProcess] Task spendb.tasks.load_from_source[14d7ea38-b1c1-4540-b9df-9d14fca22263] raised unexpected: S3ResponseError: 400 Bad Request
Traceback (most recent call last):
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(_args, *_kwargs)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
return self.run(_args, *_kwargs)
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/tasks.py", line 36, in load_from_source
source = tasks.transform_source(dataset, source_name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/etl/job.py", line 74, in wrapper
job.failed()
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/etl/job.py", line 53, in failed
self.end(Run.STATUS_FAILED)
File "/Volumes/MacintoshHD2/Users/mattbirgit/checkouts/spendb/spendb/etl/job.py", line 39, in end
self.log_handler.archive()
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/loadkit/logger.py", line 27, in archive
logfile = LogFile(self.package, name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/resource.py", line 21, in init
self.meta = ResourceMetaData(self)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/manifest.py", line 38, in init
self.manifest = resource.package.manifest
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/package.py", line 42, in manifest
self._manifest = Manifest(obj)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/manifest.py", line 14, in init
self.load()
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/manifest.py", line 17, in load
if self.object.exists():
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/store/s3.py", line 98, in exists
self._key = self.store.bucket.get_key(self._key_name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/archivekit/store/s3.py", line 34, in bucket
self._bucket = self.conn.get_bucket(self.bucket_name)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/boto/s3/connection.py", line 502, in get_bucket
return self.head_bucket(bucket_name, headers=headers)
File "/Volumes/MacintoshHD2/Users/mattbirgit/venvs/spendb/lib/python2.7/site-packages/boto/s3/connection.py", line 549, in head_bucket
response.status, response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request