Transition from S3 to Glacier without copy and delete
Opened this issue · 0 comments
nateiam commented
As a Herd Administrator, I want data to move from S3 to Glacier without having to copy the data to another bucket and delete it in the origin bucket.
Tech Note:
- Storage class transition currently operates by performing an S3 to S3 copy to a bucket with a 0d Glacier policy
- This story re-implements storage class transition to occur within the same bucket via object tagging and lifecycle policy
Acceptance Criteria
- Storage policy transitions execute according to new technical approach described above.
- Storage unit for the BData cycles through the following during transition
** Enabled - normal status prior to any archiving
** Archiving - status while archiving (tech note: this applies only to S3 object tagging, not AWS async processing that actually transitions object)
** Archived - status once archiving is complete (tech note: again applies to when tagging is complete, AWS async processing could be ongoing after this) - Storage policy transitions work only for S3_MANAGED Bucket. Other buckets including KMS will be handled as future stories
- Availability and DDL consider archived objects as "NotAvailable" (tech note: minor changes to existing logic that does this)
- Existing rule that limits transition based on BData size is removed
- Existing logic and data around additional Glacier storage is removed, including:
** Operations that move data to GLACIER_MANAGED
** Logic that checks file existence and size after archiving
Infrastructure required:
** Glacier Lifecycle Policy
** Permissions for S3 object tagging
Existing logic that does not need to be modified and does not need to be separately tested
- All Storage Policy Scheduler logic
- Check that archiving is already underway