logstash-plugins/logstash-input-s3

Doc enhancement for backup_add_prefix and backup_to_bucket

shikrallah opened this issue · 0 comments

The documentation for backup_add_prefix should mention that it does not have any effect unless the backup_to_bucket parameter is also specified.

In a related note it could be useful to mention that backup_to_bucket can be set to the same bucket that is being read from, and used in conjunction with backup_add_prefix to back up objects to a different path within the same bucket.

For instance a configuration such as the following could be used to store all objects in one bucket but avoid continually checking the older objects against the since DB to see if they had changed and/or avoid a new machine processing all old objects.

input {
s3 {
bucket => "source-bucket"
backup_to_bucket => "source-bucket"
backup_add_prefix => "processed"
delete => true
exclude_pattern => "processed"
}
}

or alternately

input {
s3 {
source => "source-bucket"
prefix => "data"
backup_to_bucket => "source-bucket"
backup_add_prefix => "processed"
delete => true
}
}

Regards,
Bill