facebookresearch/DeeperCluster

How to get YFCC100M with the 'flickr_unique_ids.npy' file?

Chen-Song opened this issue · 2 comments

Chen-Song commented

Please tell me how to use 'flickr_unique_ids.npy' file to download the YFCC100M dataset, thank you.
Look forward your reply.
Best,
SongChen

Chen-Song commented

Hi
I downloaded the dataset as the way as follows:

  1. Run "pip install awscli" to install awscli.
  2. Run "aws configure" and enter your access key and secret (available via https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key).
  3. Run "aws s3 ls s3://yahoo-webscope-i3/" to view the S3 objects for I3 - Yahoo Flickr Creative Commons 100M (14G) (Hosted on AWS).
  4. Run "aws s3 cp s3://yahoo-webscope-i3 . --recursive" to download I3 - Yahoo Flickr Creative Commons 100M (14G) (Hosted on AWS) to current directory.
    But there were a lot of failures.
    First, my aws version is 'aws-cli/1.16.234 Python/3.6.5 Linux/4.15.0-55-generic botocore/1.12.224' and in 'aws configure' command, I set the 'region name=us-east-2'. In fact, my location is Beijing, China, but when I set the region name= cn-north-1, I get an error message that the access ID is not available.
    Second, I set the 'Default output format=json'.
    Third, I run aws s3 cp s3://yahoo-webscope-i3 . --recursive, the download terminal as follows:

download: s3://yahoo-webscope-i3/WebscopeReadMe.txt to ./WebscopeReadMe.txt
download failed: s3://yahoo-webscope-i3/yfcc100m_places.bz2 to ./yfcc100m_places.bz2 Max Retries Exceeded
download failed: s3://yahoo-webscope-i3/yfcc100m_autotags.bz2 to ./yfcc100m_autotags.bz2 Max Retries Exceeded
download failed: s3://yahoo-webscope-i3/yfcc100m_exif.bz2 to ./yfcc100m_exif.bz2 Max Retries Exceeded
download failed: s3://yahoo-webscope-i3/yfcc100m_dataset.bz2 to ./yfcc100m_dataset.bz2 Connect timeout on endpoint URL: "https://yahoo-webscope-i3.s3.amazonaws.com/yfcc100m_dataset.bz2"

Please tell me what I should do to download the dataset. Look forward your reply. Thanks.
Best,
SongChen.

mathildecaron31 commented

Hi,
Thank you for your interest. Unfortunately I haven't personally downloaded the dataset so I am not sure I can help you with this issue. Maybe you could reach out directly to yahoo research group.