20110422 MALLET-EVAL PROJECT
GENERAL
This is a project for evaluating MALLET (MAchine Learning for
LanguagE Toolkit). MALLET's binary and source codes are not included,
you can check out them from this site:
http://mallet.cs.umass.edu/
This distribution only contains sample annotation data and scripts for
converting, importing and evaluating. The articles in the two corpora are
not included here for copyright reasons. That is why you need their cds
for building the complete data sets.
We provide two sample corpora: Penn Treebank Sample (5% fragment of Penn
Treebank) and HIT CIR LTP Corpora Sample (10% fragemnt of the whole
Corpora)
http://web.mit.edu/course/6/6.863/OldFiles/share/data/corpora/treebank/
http://ir.hit.edu.cn/demo/ltp/Sharing_Plan.htm
BUILDING THE TRAIN AND TEST DATA FILES
In order to obtain the data files you need to perform three steps:
1. Get a local copy of the mallet-eval repository with this command:
hg clone https://mallet-eval.googlecode.com/hg/ mallet-eval
2. Set up $MALLET_HOME enviroment: export MALLET_HOME=/path/to/mallet/
3. Train and test with provided Chunking, POS Tagging and Named Entity
Recognition data (chunking/ pos-tagging/ ner/)
4a. (Chunking) ./conlleval < chunking/conlleval.out
4b. (POS-Tagging) cd pos-tagging | ./verify.py
4c. (Named Entity Recognition) ./chn-conlleval < ner/conlleval.out
and the results are:
processed 47377 tokens with 23852 phrases; found: 23682 phrases; correct: 21441.
accuracy: 93.97%; precision: 90.54%; recall: 89.89%; FB1: 90.21
ADJP: precision: 72.35%; recall: 63.93%; FB1: 67.88 387
ADVP: precision: 78.61%; recall: 75.98%; FB1: 77.28 837
CONJP: precision: 40.00%; recall: 44.44%; FB1: 42.11 10
INTJ: precision: 50.00%; recall: 50.00%; FB1: 50.00 2
LST: precision: 0.00%; recall: 0.00%; FB1: 0.00 2
NP: precision: 90.05%; recall: 89.57%; FB1: 89.81 12355
PP: precision: 94.97%; recall: 96.88%; FB1: 95.92 4908
PRT: precision: 71.84%; recall: 69.81%; FB1: 70.81 103
SBAR: precision: 89.01%; recall: 78.69%; FB1: 83.53 473
VP: precision: 91.55%; recall: 90.51%; FB1: 91.03 4605
DATA FORMAT
The data files contain one word per line. Empty lines have been used
for marking sentence boundaries and a line containing the keyword
-DOCSTART- has been added to the beginning of each article in order
to mark article boundaries. Each non-empty line contains the following
tokens:
1. the current word
2. the lemma of the word (German only)
3. the part-of-speech (POS) tag generated by a tagger
4. the chunk tag generated by a text chunker
5. the named entity tag given by human annotators
The tagger and chunker for English are roughly similar to the
ones used in the memory-based shallow parser demo available at
http://ilk.uvt.nl/ German POS and chunk information has been
generated by the Treetagger from the University of Stuttgart:
http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/
In order to simulate a real natural language processing
environment, the POS tags and chunk tags have not been checked.
This means that they will contain errors. If you have access to
annotation software with a performance that is superior to this,
you may replace these tags by yours.
The chunk tags and the named entity tags use the IOB1 format. This
means that in general words inside entity receive the tag I-TYPE
to denote that they are Inside an entity of type TYPE. Whenever
two entities of the same type immediately follow each other, the
first word of the second entity will receive tag B-TYPE rather than
I-TYPE in order to show that a new entity starts at that word.
The raw data has the same format as the training and test material
but the final column has been ommitted. There are word lists for
English (extracted from the training data), German (extracted from
the training data), and Dutch in the directory lists. Probably you
can use the Dutch person names (PER) for English data as well. Feel
free to use any other external data sources that you might have
access to.
Max Lv <lch@fudan.edu.cn>