Add 'requester pays' options for data downloads
Opened this issue · 0 comments
chadokruse commented
- Create new bucket and turn requester pays option to 'on' (docs)
- Add additional step to deploy scripts to also send data to requester pays bucket
- Detailed README with instructions (including limitations of dataset)
Size of files (gzipped):
1.5GB Normalized filings (2011-current)
0.2GB Grants (2011-current)
Storage cost negligible (only $0.05 / mo...$0.026 / GB)
Network costs and egress fees are the concern (fee schedule)
Alternative:
-
Use G-Suite storage (API docs)
Q: What are the consequences if a developer constantly hammers the endpoint(s) programmatically? -
Use a cloud function to disallow programmatic access. Would allow good actors access, but prevent runaway costs from programmatic hammering of endpoint.