Word Segmentation and Artificial Languages
Makefile
use this to run an experiment
by default, the Unigram grammar is used on the first random hawaiian corpus
to run on a different version (there are currently 4 sets, there is nothing
special about them, they were just generated by running Robert's scripts 4
times), use
make SET=0[1234]
to run on a different language (right now, there only is Berber in addition)
make LANGUAGE=berber
to keep the model as simple as possible, the base-distribution is assumed to
be fixed, that is, words are generated assuming a uniform phoneme distribu-
tion and a constant stopping probability of 0.5, although hyper-parameter
inference is performed to get around having to set arbitrary hyper-parameters
(as of now, it uses the original DP-model instead of the slightly more expres-
-sive PYP model)
runs/
where experiments are put
there are two folders for every experiment, a Tmp-folder which contains actual
data (inputs, outputs, tracefiles) for different runs of the same experiment,
and an Eval-folder that only contains a "summary"-file which gives a single
score
experiments are referred to by the language plus the grammar name, e.g.
runs/hawaiian_unigram(Tmp|Eval)
to distinguish different corpus versions, each file has "s0[1234]" in its
name. To illustrate,
runs/berber_unigramEval/r00_Gunigram_n1000_w1_b1_g100_h0.01_R-1_s01.trscore
runs/berber_unigramEval/r00_Gunigram_n1000_w1_b1_g100_h0.01_R-1_s02.trscore
are the score-files for the unigram-model on two different random berber
corpora
data/
contains test-corpora, named as corpus_<language>_<set>.txt, e.g.
corpus_hawaiian_01.txt
corpus_berber_04.txt
scripts/
contains scripts to generate artificial corpora / analyse their properties
corpus/
scripts to generate corpora
analysis/
scripts to perform posterior analysis (input / output projections)
use as follows:
cat runs/hawaiian_unigramTmp/r00_Gunigram_n1000_w1_b1_g100_h0.01_R-1_s02_fold0*.trsws | python scripts/analysis/posteriorOutputprojections.py runs/hawaiian_unigramTmp/AGgold_02.txt
ambiguity/
scripts to calculate segmentation ambiguity
dictionaries/
contains dictionaries generated by Robert's MaxEnt-grammar-generator
prog_seg/
contains scripts required by the Adaptor Grammar evaluation
py-cfg/
contains source code for Adaptor Grammars (use Makefile to build)
only needs to be compiled once