LargeObjectFile should support cleanup in case of upload error
majewsky opened this issue · 1 comments
majewsky commented
The internal struct implementing the LargeObjectFile
interface keeps track of which segments it has uploaded since {Dynamic,Static}LargeObjectCreate
. When the upload fails (either during PUT of another segment or PUT of the manifest), it is useful to know which segments were already uploaded in order to perform cleanup.
Since LargeObjectFile does not have a method to expose its segments, one can only perform this cleanup by listing the uploaded segments, i.e.
names, err := conn.ObjectNameAll(segmentContainer,
&swift.ObjectsOpts{Prefix: segmentPrefix},
)
... handle err ...
result, err := conn.BulkDelete(segmentContainer, names)
However, this is prone to eventual consistency issues: The segments might not yet be visible in the container listing. I see two possible options:
- Add a method like
SegmentNames() []string
to theLargeObjectFile
interface. - Add a method
Cleanup() error
orReset() error
to theLargeObjectFile
interface that deletes all segments which were uploaded since{Dynamic,Static}LargeObjectCreate
.
ncw commented
I like your Cleanup() error
proposal. Fancy sending a PR?