cartography-cncf/cartography

AWS: Make fetches concurrent: ECR and S3

achantavy opened this issue · 0 comments

Discussed in #1201

Description:

Describe your idea. Please be detailed. If a feature request, please
describe the desired behavior, what scenario it enables, and how it
would be used.

As a proof of concept for parallel gets, implement async fetching for AWS ECR and S3. In initial testing we've seen speedup improvements from 15 minutes to 2 minutes so this is worth pursuing.

Ensure that

  • The solution is easy to apply for other modules
  • We have a plan on how to handle rate limits being exceeded - e.g. with backoff + retry + giveup
  • We have documentation on how new developers can employ the pattern for their own intel modules.