The Signal Media One-Million News Articles Dataset dataset by Signal Media was released to facilitate conducting research on news articles. It can be used for submissions to the NewsIR'16 workshop, but it is intended to serve the community for research on news retrieval in general.
The articles of the dataset were originally collected by Moreover Technologies (one of Signal's content providers) from a variety of news sources for a period of 1 month (1-30 September 2015). It contains 1 million articles that are mainly English, but they also include non-English and multi-lingual articles. Sources of these articles include major ones, such as Reuters, in addition to local news sources and blogs.
To obtain the dataset, follow the download link here.
Elasticsearch is a powerful distributed RESTful search engine that can be used to store and index large amounts of data. At Signal, we use Elasticsearch to handle most of our search requests.
- Download Elasticsearch and unzip.
- Run
bin/elasticsearchon Unix orbin/elasticsearch.baton Windows. - Run
curl -X GET http://localhost:9200/
At this point, Elasticsearch should be running locally on port 9200. More information about Elasticsearch can be found at their GitHub page.
We advise that you use a tool to interact with Elasticsearch. Here are a few good ones:
In order to store articles, you need to create an index. First, create an articles index:
curl -X PUT 'http://localhost:9200/articles'or in Sense:
PUT articlesTo index the million articles into Elasticsearch using python, first install Requests:
pip install requestsThen run:
python index_articles.py http://localhost:9200/articles ./million.jsonlThe term and document frequencies are also available using these links. These values were calculated after routine tokenisation and stop-word removal.
These files are in edn format.
A script to convert the Signal Media One-Million News Articles Dataset to TREC format. The TREC format allows researchers to index the dataset using popular Information Retrieval platforms such as http://terrier.org
After obtaining the dataset through this form http://goo.gl/forms/5i4KldoWIX, you can extract the JSONL file from the the downloaded Gzip file Then you run the script like this
python convert-to-trec.py -i <path to signalmedia-1m.jsonl> -o <path to your outputfile>We recommend using the terrier.properties file included in this repository to index the dataset with Terrier. In your Terrier etc folder, add a text file "signal.spec" with one line containing the path to the file you created above (The TREC formatted dataset)