AWS: Make fetches concurrent: ECR and S3
achantavy opened this issue · 0 comments
achantavy commented
Discussed in #1201
Description:
Describe your idea. Please be detailed. If a feature request, please
describe the desired behavior, what scenario it enables, and how it
would be used.
As a proof of concept for parallel gets, implement async fetching for AWS ECR and S3. In initial testing we've seen speedup improvements from 15 minutes to 2 minutes so this is worth pursuing.
Ensure that
- The solution is easy to apply for other modules
- We have a plan on how to handle rate limits being exceeded - e.g. with backoff + retry + giveup
- We have documentation on how new developers can employ the pattern for their own intel modules.