Metadata overwritten and needs to be reconsolidated.
Closed this issue · 2 comments
It looks like the shot only has one group...
But the other sources are there.
Need to make sure the ingestion pipeline reconsolidates the metadata if a new group is written.
Can we change the bucket policy so that the objects are not read-only for our pipeline for example? I think the only sensible way past this issue is to reconsolidate once the object is uploaded to S3 @NathanCummings
Currently consolidating all the data that exists already - to get around this issue in the future we can either run the same script that I have done now (takes some time ~ 1 day for 19,000 shots) or reconsolidate manually ourselves (i.e. without using zarr).
Another option is to consider Icechunk, as Nathan mentioned it may remove the need for consolidation as it is already rather quick loading from S3.
Will investigate Icechunk as an alternative first, as this aligns with #58 as well.