Add CommonCrawl 2019 Archives
Closed this issue · 1 comments
tja commented
Our URL database only has CommonCrawl archives starting from 2020. We should add earlier archives over time, starting with the 2019 archives:
- https://commoncrawl.org/2019/01/january-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/03/february-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/04/march-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/04/april-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/05/may-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/07/june-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/07/july-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/08/august-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/09/september-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/10/october-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/11/november-2019-crawl-archive-now-available/
- https://commoncrawl.org/2019/12/december-2019-crawl-archive-now-available/
These should be integrated into our database.
tja commented
All CommonCrawl 2019 archives have been added.