ncw/swift

LargeObjectFile should support cleanup in case of upload error

majewsky opened this issue · 1 comments

The internal struct implementing the LargeObjectFile interface keeps track of which segments it has uploaded since {Dynamic,Static}LargeObjectCreate. When the upload fails (either during PUT of another segment or PUT of the manifest), it is useful to know which segments were already uploaded in order to perform cleanup.

Since LargeObjectFile does not have a method to expose its segments, one can only perform this cleanup by listing the uploaded segments, i.e.

names, err := conn.ObjectNameAll(segmentContainer,
  &swift.ObjectsOpts{Prefix: segmentPrefix},
)
... handle err ...
result, err := conn.BulkDelete(segmentContainer, names)

However, this is prone to eventual consistency issues: The segments might not yet be visible in the container listing. I see two possible options:

  1. Add a method like SegmentNames() []string to the LargeObjectFile interface.
  2. Add a method Cleanup() error or Reset() error to the LargeObjectFile interface that deletes all segments which were uploaded since {Dynamic,Static}LargeObjectCreate.
ncw commented

I like your Cleanup() error proposal. Fancy sending a PR?