reg-viz/reg-suit

[reg-publish-s3-plugin] Downloading objects never finishes when there are more than 1000 objects in the download

fukumasuya opened this issue · 0 comments

Describe the bug

It seems reg-publish-s3-plugin does not list all objects in the downloading process when there are more than 1000 objects.
The plugin passes Marker parameter to ListObjectsV2Command to get all objects recursively.
But ListObjectsV2Command does not support Marker and ignores it. So ListObjectsV2Command is called infinitely..
We should use ContinuationToken instead.

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-client-s3/Interface/ListObjectsV2CommandInput/

if (lastKey) {
options.Marker = lastKey;
}
return new Promise<ObjectListResult>((resolve, reject) => {
this._s3client
.send(new ListObjectsV2Command(options))

Reproduced step

Steps to reproduce the behavior:

  1. Prepare an environment that has more than 1000 objects in a S3 bucket
  2. Run reg-suit with reg-publish-s3-plugin

Expected behavior

Downloading items finishes.

Actual behavior

Downloading items never finishes.