grantmakers/grantmakers.github.io

Add 'requester pays' options for data downloads

Opened this issue · 0 comments

  • Create new bucket and turn requester pays option to 'on' (docs)
  • Add additional step to deploy scripts to also send data to requester pays bucket
  • Detailed README with instructions (including limitations of dataset)

Size of files (gzipped):
1.5GB Normalized filings (2011-current)
0.2GB Grants (2011-current)

Storage cost negligible (only $0.05 / mo...$0.026 / GB)
Network costs and egress fees are the concern (fee schedule)


Alternative:

  1. Use G-Suite storage (API docs)
    Q: What are the consequences if a developer constantly hammers the endpoint(s) programmatically?

  2. Use a cloud function to disallow programmatic access. Would allow good actors access, but prevent runaway costs from programmatic hammering of endpoint.