FINRAOS/herd

Transition from S3 to Glacier without copy and delete

Opened this issue · 0 comments

As a Herd Administrator, I want data to move from S3 to Glacier without having to copy the data to another bucket and delete it in the origin bucket.

Tech Note:

  • Storage class transition currently operates by performing an S3 to S3 copy to a bucket with a 0d Glacier policy
  • This story re-implements storage class transition to occur within the same bucket via object tagging and lifecycle policy

Acceptance Criteria

  • Storage policy transitions execute according to new technical approach described above.
  • Storage unit for the BData cycles through the following during transition
    ** Enabled - normal status prior to any archiving
    ** Archiving - status while archiving (tech note: this applies only to S3 object tagging, not AWS async processing that actually transitions object)
    ** Archived - status once archiving is complete (tech note: again applies to when tagging is complete, AWS async processing could be ongoing after this)
  • Storage policy transitions work only for S3_MANAGED Bucket. Other buckets including KMS will be handled as future stories
  • Availability and DDL consider archived objects as "NotAvailable" (tech note: minor changes to existing logic that does this)
  • Existing rule that limits transition based on BData size is removed
  • Existing logic and data around additional Glacier storage is removed, including:
    ** Operations that move data to GLACIER_MANAGED
    ** Logic that checks file existence and size after archiving

Infrastructure required:
** Glacier Lifecycle Policy
** Permissions for S3 object tagging

Existing logic that does not need to be modified and does not need to be separately tested

  • All Storage Policy Scheduler logic
  • Check that archiving is already underway