Support range queries when fetching snapshots
Closed this issue · 0 comments
lemmih commented
Issue summary
We currently ignore range queries when downloading snapshots from CloudFlare. Without range queries, we cannot resume downloads.
Tasks:
- Set up local development environment with
wrangler dev
. - Download the latest calibnet snapshot from your local deployment. (This will use the
forest-archive-dev
bucket rather than the productionforest-archive
bucket). - Confirm that range queries do not work.
- Extract range parameters from the HTTP request (see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range).
- Pass range parameters to R2.get (see: https://developers.cloudflare.com/r2/api/workers/workers-api-reference/#ranged-reads)
- Confirm that resume works.
- Confirm that downloading multiple chunks in parallel works (use
aria2c -x5
).
This is the R2.get
method that needs range options:
Other information and links
R2 worker API: https://developers.cloudflare.com/r2/api/workers/workers-api-reference/
Logging headers to console: https://developers.cloudflare.com/workers/examples/logging-headers/
JavaScript library for parsing range headers: https://www.npmjs.com/package/range-parser