s3.listFiles does not manage truncation
shn-amn opened this issue · 2 comments
shn-amn commented
In response to a ListObjectsV2Request, if a bucket contains too many objects (more than a 1000 in my experience), AWS automatically truncates the response to a reasonable size (1000, as before) and sends along with it a continuation token to send the next request.
This behaviour is completely ignored by s3.listFiles, leading to an obvious bug while listing crowded buckets.
kiambogo commented
Looks like auto-pagination is supported in v2 of the SDK, so I think its best to focus on migrating the S3 functions to V2 of the SDK instead of adding in support for manual pagination.
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-pagination.html