/techreview_arxiv_scrape

Primary LanguageJupyter Notebook

techreview_arxiv_scrape

The data and partial code behind the story "We analyzed 16,625 papers to figure out where AI is headed next" on MIT Technology Review.

Files

  • arxivscraper.py: A modified version of Mahdisadjadi/arxivscraper
  • arxivscraper_all_cs_ai.ipynb: The code for scraping metadata from the arXiv
  • cs_ai_through_2018-11-18.csv: The metadata for all scientific papers published to the AI section of the arXiv through November 18, 2018
  • arxivscraper_ai_only_analysis.ipynb: The code for counting all of the words and ngrams by year, and for calculating those with the greatest losses and gains

Data columns

  • id: arXiv paper id
  • title: paper title
  • categories: the arXiv categories under which the paper was filed
  • abstract: paper abstract
  • doi: digitial object identifier
  • created: date of paper's upload
  • updated: date of paper's latest update
  • authors: paper authors last names
  • year: year and month of paper's upload

Source