/nlp_research

The Research Spider for Anthology. This toolkit can automatically grab the papers information by given keywords in title. You can set the search params: the conference, publication date and the keywords in title . Then it will automatically save these papers' title, authors, download link, Google Scholar cited number and abstracts information in a Excel file.

Primary LanguagePython

Spider

A toolkit for NLP researchers, which used to automatically grab papers' info.

Research Spider for Anthology v0.5

Introduction

This toolkit can automatically grab the papers information by given keywords in title.

You can set the search params: the conference, publication date and the keywords in title . Then it will automatically save these papers' title, authors, download link, Google Scholar cited number and abstracts information in a 'Excel' format file.

Usage

  • Parameters and Setting:

    Config parameters in the script research_spider.py

    Parameters Setting Description
    published year years = (2013, 2016) years from 2013 to 2016
    keywords keywords = ['lexicon', 'dictionary', 'lexical'] keywords
    conferences events = ['ACL', 'CL', 'COLING', 'EACL', 'EMNLP', 'LREC', 'NAACL'] conferences

  • configure file You can config the spider mode and google scholar website url in the configure file conf/conf.ini.

    • This toolkit support two mode, selenium and requests, and default mode is selenium.
    • We can not access Google Scholar website in China, so i use a mirror website.

The selenium mode is recommended for use. Please download chromedriver and install the chrome browser to use the selenium mode. You can download the chromedriver from mirror website. Make sure your chromedriver version is consistent with your chrome browser.

ChromeDriver 76.0.3809.12 (2019-06-07)---------Supports Chrome version 76
ChromeDriver 75.0.3770.8 (2019-04-29)---------Supports Chrome version 75

  • Usage

    python3 research_spider.py

Contibutor

Author: wang leyi

Email: leyiwang.cn@gmail.com

Blog:https://leyiwang.github.io/

Change History

Date: Last update 2019-08-22 Description:

  1. Refactor code to adapt changes of Anthology
  2. Add the selenium mode to accelerate the efficiency and improving the stability of spider

Date: Update 2019-04-16 Description: Refactor code to adapt to Anthology site style changes

Date: Update 2017-05-02 Description: Fix the invalid bug for the second bit of 0 for the year.

Data: Update 2016-12-22

Date: Update 2016-11-5