igrigorik/gharchive.org

Range download is not working

bhargav-velisetti opened this issue · 1 comments

Screenshot 2024-05-05 173328

Could you please confirm If I can download multiple files rather than single file for each attempt.

Thanks

When you look at the cron job that creates the zip files, I assume that you have to download them individually and that this is just pseudo code

5 * * * * /bin/bash -l -c 'cd /home/igrigorik/githubarchive.org/crawler/data && gzip -9 `date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json && gsutil -h "Content-Type: application/gzip" cp -a public-read `date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json.gz gs://data.gharchive.org/' >> /home/igrigorik/githubarchive.org/crawler/data/sync.log 2>&1