Package/compress model artifacts on COS
Closed this issue · 4 comments
The [checkpoint] assets are currently stored in individual files on COS, e.g.
http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/audioset/vggish_model.ckpt
http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/audioset/vggish_pca_params.npz. The individual files should be packaged and compressed in a tar.gz archive, following the approach used by other models. Refer to https://github.com/IBM/MAX-Object-Detector/blob/master/Dockerfile for an example.
Benefits:
- Only a single file needs to be downloaded (less work)
- Artifacts are compressed, potentially reducing the time it takes to download them
@kmh4321 you can probably duplicate this PR (IBM/MAX-Audio-Embedding-Generator#4) on this repository.
@bdwyer2 thank you for this. Should I re-compress all 4 files I need (the 3 that overlap with your PR + 1 extra file) or re-use the .tar.gz file which is being used by the embedding generator and have another wget for the classifier weights? The latter eliminates data duplication on COS and would not require changing weights at two places in the future, but would involve an extra wget command.
Didn't realize that you had more files, I guess you should zip everything together.
It'll be cleaner to add all your required files to 1 archive. We should have an archive per model for consistency and to keep things cleanly separated.