- Note this can be slow, e.g. 180M files, 69G, took 8h to list with 20mbps cap, zipped to 5GB.
-
Download AzCopy from Microsoft - [https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10]
-
Generate SAS token to access your blob storage, it needs [x] List
-
On huge blob storage this can take more thatn 24h make expiry a week.
-
export to env.
export Azure_Sas_Url=""
-
-
Run the azcopy list, this can take long if storage big (Terrabytes, Millions of files)
time ./azcopy-macos list "${Azure_Sas_Url}" --machine-readable --output-type json --properties "LastModifiedTime;VersionId;BlobType;BlobAccessTier;ContentType;ContentEncoding" --cap-mbps 100 --running-tally | gzip > ~/az_list_blob-$(date +%F).json.gz -
Check progress with
ls -lh ~/az_list_blob-* time cat ~/az_list_blob-$(date +%F).json.gz|gunzip| wc -l | sed -e "s/[[:space:]][[:space:]]/ /g" | sed -e 's/^[[:space:]]//g' | numfmt --to=si
python3 ./report_blob/report_blob.py -i ~/az_list_blob-$(date +%F).json.gz > report-$(date +%F).txt
| container | cnt | bytesGB | bytes | bytes_max | bytes_min | cnt_zerobyte |
|---|---|---|---|---|---|---|
| images1 | 27 | 0 GB | 62,009 b | 0.0 MB(max) | 519 b(min) | 0 0b |
| images2 | 1 | 0 GB | 169,490 b | 0.2 MB(max) | 169490 b(min) | 0 0b |
| images3 | 24 | 0 GB | 818,432 b | 0.1 MB(max) | 1022 b(min) | 0 0b |
| images4 | 4 | 0 GB | 2,102,104 b | 1.0 MB(max) | 33543 b(min) | 0 0b |
| report1 | 161 | 0 GB | 2,604,174 b | 0.2 MB(max) | 5164 b(min) | 0 0b |
| images5 | 703 | 0 GB | 3,241,566 b | 0.0 MB(max) | 281 b(min) | 0 0b |
| images6 | 656 | 0 GB | 8,531,504 b | 0.7 MB(max) | 823 b(min) | 0 0b |
| report2 | 267 | 0 GB | 13,602,358 b | 0.1 MB(max) | 5126 b(min) | 0 0b |
| errors1 | 173 | 0 GB | 14,050,777 b | 1.2 MB(max) | 5145 b(min) | 0 0b |
| users1 | 220 | 0 GB | 18,230,688 b | 1.0 MB(max) | 9278 b(min) | 0 0b |
| error2 | 305 | 0 GB | 27,510,955 b | 2.5 MB(max) | 5173 b(min) | 0 0b |
| images7 | 3,198 | 0 GB | 33,570,690 b | 0.3 MB(max) | 305 b(min) | 0 0b |
| images8 | 8,227 | 0 GB | 44,302,581 b | 0.1 MB(max) | 136 b(min) | 0 0b |
| errors3 | 12,622 | 0 GB | 74,007,047 b | 0.3 MB(max) | 5423 b(min) | 0 0b |
| errors4 | 45,000 | 0 GB | 166,897,802 b | 3.1 MB(max) | 406 b(min) | 0 0b |
| images8 | 16,783 | 0 GB | 211,683,261 b | 0.6 MB(max) | 303 b(min) | 0 0b |
| images9 | 4,629 | 0 GB | 308,341,642 b | 1.4 MB(max) | 96 b(min) | 0 0b |
| images9 | 9,649 | 1 GB | 740,536,779 b | 4.8 MB(max) | 0 b(min) | 1 0b |
| userimagesA | 15,000 | 1 GB | 749,941,609 b | 2.7 MB(max) | 84 b(min) | 0 0b |
| userimagesB | 61,198 | 1 GB | 857,806,656 b | 1.1 MB(max) | 305 b(min) | 0 0b |
| userimagesC | 55,000 | 1 GB | 1,047,880,398 b | 0.5 MB(max) | 279 b(min) | 0 0b |
| userimagesD | 54,697 | 1 GB | 1,083,548,739 b | 0.9 MB(max) | 269 b(min) | 0 0b |
| userimagesE | 21,077 | 1 GB | 1,090,520,412 b | 1.6 MB(max) | 267 b(min) | 0 0b |
| export-reports | 15,676 | 1 GB | 1,136,073,132 b | 17.1 MB(max) | 5183 b(min) | 0 0b |
| shippingreports | 21,290 | 1 GB | 1,208,945,757 b | 0.1 MB(max) | 122 b(min) | 0 0b |
| generalreports | 36,995 | 1 GB | 1,512,532,337 b | 19.6 MB(max) | 5351 b(min) | 0 0b |
| images10 | 70,000 | 2 GB | 2,301,267,403 b | 3.2 MB(max) | 0 b(min) | 1 0b |
| images11 | 55,000 | 5 GB | 5,653,622,358 b | 20.0 MB(max) | 0 b(min) | 38 0b |
| images12 | 70,000 | 6 GB | 6,697,986,128 b | 1.7 MB(max) | 0 b(min) | 1 0b |
| images13 | 60,000 | 7 GB | 7,643,115,428 b | 2.7 MB(max) | 99 b(min) | 0 0b |
| images14 | 70,000 | 10 GB | 10,319,261,396 b | 1.5 MB(max) | 0 b(min) | 2 0b |
| images15 | 65,000 | 17 GB | 18,021,096,921 b | 1.6 MB(max) | 97 b(min) | 0 0b |
| --- blob-totals --- | 1,000,000 | 498 GB | 534,953,996,770 b | 113.8 MB(max) | 0 b(min) | 43 0b |