#################################################################################################
# Kaggle "The Allen AI Science Challenge" competition
# Oct 2015 - Feb 2016
# Kaggle username: "Cardal"
#################################################################################################
This file explains how to run the ai2_Cardal.py code, which generates my solution for the
Kaggle AI2 competition.
1. System and Dependencies
The code is written in python. I used Python 2.7.7 on Windows 7 Professional (8GB RAM) for
most of the executions.
The following python libraries are used by the script:
numpy (1.8.1), scipy (0.14.0), pandas (0.14.0), sklearn (0.15.2), nltk (3.0.5)
Since I couldn't get PyLucene to work on my system, I ran the parts that use Lucene on
a Linux machine (16GB RAM) with Python 2.6.6 and the following libraries:
numpy (1.8.1), scipy (0.14.0), pandas (0.15.2), sklearn (0.16.0), nltk (3.1), lucene (pylucene-4.7.2-1)
2. Files
Extract the archive to a directory on your system.
Copy the test set file to the "input" folder and change the settings as described below.
3. Resources
The model uses several resources that were approved in the competition forum (in
the "External Data Repository" thread).
All files are supplied in the "corpus" folder, except the Wikipedia dumps and the pdf
text books from www.ck12.org (see below).
(a) AI2 - data downloaded from the AI2 web-site: http://allenai.org/data.html
I extracted all the text from these datasets and organized it into a single
corpus file at corpus/AI2_data/ai2_corpus.txt
(b) CK12 - I downloaded two types of resources from the http://www.ck12.org:
First, I used the HTML files in the EPUB file "Concepts_b_v8_vdt.epub".
Second, I downloaded the science-related books in pdf format ("FlexBooks"), and converted them
into text files using MS Word (with the "Save as" option). The converted files are
under corpus/CK12 with the ".text" suffix; the original pdf books are quite large, so they
aren't included here (I kept the original name, only replaced the suffix).
I prepared four corpus files from the CK12 books:
corpus/CK12/OEBPS/ck12.txt - based on the HTML files in "Concepts_b_v8_vdt.epub";
corpus/CK12/OEBPS/ck12_paragraphs.txt - as above, but using a different criterion
for splitting the text into sections (paragraphs);
corpus/CK12/ck12_text.txt - based on the pdf files;
corpus/CK12/ck12_text_sentences.txt - as above, but using a different criterion
for splitting the text into sections (sentences).
(c) Quizlet - study cards from quizlet.com. I parsed the cards (e.g., in order to remove
wrong answers from multiple-choice questions), and manually editted some of them (e.g.,
to fix format problems). The full corpus is in corpus/quizlet/quizlet_corpus.txt
(d) Saylor and OpenStax - books in pdf format from Saylor Academy (http://www.saylor.org/books/)
and OpenStax College (https://www.openstaxcollege.org/books).
I extracted the text from the pdf files using MS Word or (for large files) the pdfminer tool ("pdf2txt.py").
The full corpus is in corpus/Saylor/saylor_text.txt
(e) SimpleWiki - I downloaded the SimpleWiki dump file "simplewiki-20151102-pages-articles.xml" (496MB)
from https://dumps.wikimedia.org/simplewiki/20151102/, and constructed three corpus files by extracting
some sections from subsets of the pages. The files are under corpus/simplewiki:
simplewiki_1.0000_0.0500_0_5_True_True_True_corpus.txt (117MB)
simplewiki_1.0000_0.1000_0_3_True_True_False_corpus.txt (114MB)
simplewiki_1.0000_0.0100_0_3_True_True_False_pn59342_corpus.txt (4.4MB)
(f) StudyStack - study cards from http://www.studystack.com. I parsed the cards (e.g., to
remove sentences marked as False), and saved four versions under corpus/StudyStack, each
containing a different subset of the cards:
studystack_corpus.txt (72MB)
studystack_corpus2.txt (4MB)
studystack_corpus3.txt (9MB)
studystack_corpus4.txt (17MB)
(g) UtahOER - Utah Science Open Educational Resources textbooks downloaded from
http://www.schools.utah.gov/CURR/science/OER.aspx. I extracted the text from the pdf
files using the pdfminer tool. The entire corpus is in corpus/UtahOER/oer_text.txt
(h) Wiki - I downloaded the Wiki dump file "enwiki-20150901-pages-articles.xml" (50.6GB)
from https://dumps.wikimedia.org/enwiki/20150901/. I created two corpus files, containing
different sections from subsets of the pages, under corpus/wiki:
wiki_1.0000_0.0200_0_5_True_True_False_corpus.txt (161MB)
wiki_0.5000_0.1000_0_5_True_True_False_pn59342_corpus.txt (33MB)
(i) Wikibooks - I downloaded the WikiBooks dump file "enwikibooks-20151102-pages-articles.xml" (560MB)
from https://dumps.wikimedia.org/enwikibooks/20151102/, and created a corpus file with
some sections from a subset of the pages:
wikibooks_1.0000_0.0200_0_10_True_True_False_corpus.txt (126MB)
4. Settings
The SETTINGS.json file contains the following parameters:
"BASE_DIR" - the directory into which you extracted the archive
"INPUT_DIR" - directory with input files (training, validation, test sets)
"CORPUS_DIR" - directory with corpus files, organized in sub-folders
"SUBMISSION_DIR" - directory for the submission (solution) files
"TRAINING_FILE" - training set file
"VALIDATION_FILE" - validation set file
"TESTING_FILE" - test set file
=> IMPORTANT: Please change "BASE_DIR" and "TESTING_FILE" to the actual names of the directory in
which you extracted the files and the test set file, respectively. The other
settings should probably not be changed.
5. Execution
The entire code is in the "ai2_Cardal.py" script. There are two phases for the execution.
Phase A: Feature preparation
In this phase, the script computes features for all questions and answers. There are 28 sets
of features. Set 0 are "basic", in the sense that they are extracted from the questions
and answers themselves (in the entire training set), without any additional corpus.
Each of the other 27 feature sets consists of one or two features that are computed
by searching for words from the questions and answers in one or more corpora.
Features 1-20 use my search functions, and some of them are quite slow.
Features 21-27 use PyLucene with the index files supplied in corpus/lucene_idx[1-7] (if you
remove these files, the code will prepare the indexes from the corpus files).
In order to speed-up the total runtime by utilizing multiple CPUs, each set of features can
be computed separately (and in parallel), by running: "python ai2_Cardal.py prep [0-27]"
Thus, you should run:
python ai2_Cardal.py prep 0
python ai2_Cardal.py prep 1
...and so on until:
python ai2_Cardal.py prep 27
Some features take minutes to calculate, others could take several hours or more.
The model archive contains features computed for the training and validation sets (in the
folders "funcs_train" and "funcs_valid", respectively), so running "prep" as above will
calculate the features only for the test set. In order to re-compute features, simply
delete the relevant files from the "funcs_" directories.
Note: In order to save computations, the code checks whether a question in the test set
appears in the validation set; if so, it copies the scores each answer attained,
instead of re-computing them. This assumes that the common questions share the same
ID in both the validation and test files.
Phase B: Model construction
After all the features have been calculated, you should train the model and build the
solution using:
python ai2_Cardal.py run
(Note: This will also compute any missing features that haven't been prepared in the
previous phase)
The "run" command first performs cross-validation on the training set and reports the
scores (AUC and accuracy). It then trains the model on the complete training data and
predicts the answers for both the validation and test sets. These solutions are
written to files in the "submission" folder.
The training cross-validation scores should be as follows (by default, the code runs
only one CV iteration):
CV scores (mean = 0.83054): 0.83054
CV accuracy (mean = 0.61600): 0.61600
The model's predictions for the validation set should be:
validation predictions (8132):
102501: C (1.34e-01, 6.04e-02, 6.57e-01, 6.17e-02)
102502: D (1.18e-01, 2.73e-01, 9.23e-02, 4.63e-01)
...
102529: B (7.29e-02, 7.93e-01, 1.11e-01, 6.46e-02)
102530: A (3.88e-01, 1.36e-01, 1.17e-01, 2.93e-01)
The solution generated for the validation set in the "submission" directory should be
identical to the one supplied in the archive (submission/ai2_cardal_validation_20160204_1846.csv).