The extracted preprocessed feature is super big

Question

The extracted preprocessed feature is super big

ElegantLin opened this issue 3 years ago · 6 comments

Hi, I am extracting the pre-processed feature and it is super big, which is over several TB. The extraction command I use is tar zxvf **.tar.gz. Could you please help me with this?

Thanks!

Answer 1 · 2022-04-20T20:01:31.000Z

Hi,
The data shouldn't be that big, the total download size is around 130GB. The extraction might take some time, but the command looks right.

Answer 2 · 2022-04-21T15:33:31.000Z

After extraction, I found the largest file is data.mdb, which is 700GB for every file. May I ask whether it is really that big?

Thanks!

Answer 3 · 2022-04-23T02:17:36.000Z

Hi, can you do du -sh . inside the extracted feature dir? It shouldn't be that big, those are precomputed image features.

Answer 4 · 2022-04-24T17:01:54.000Z

Here is the preview of the folder. I cannot extract all of them so I cannot execute du -sh . inside the extracted feature dir.

I think the size is much more different than yours. Could you please tell me the actual size of the feature before compressing them?

Thanks for your help!

Answer 5 · 2022-04-24T17:14:12.000Z

I tried to extract the features at my end, there seems to be an issue with extracting mdb files. I would suggest you to try and do data preprocessing at your end (as given in the README in this section), while we try to fix this issue.

Answer 6 · 2022-04-24T17:16:57.000Z

Thanks a lot! Please let me know when you finish fixing it.