A small Clojure application which extends the Berkeley Aligner to support incremental alignment, e.g. using an already-trained alignment model to generate alignments on new data without having to retrain the model.
The easiest option is to download the jar file and run it from the command line (see below).
Code-tweakers can also clone this repo and use the Leningen toolbox (e.g. lein run
) to try out variations. Note that you may hit dependency issues - I installed the Berkeley Aligner jar in my local lib as per these instructions to get it in the classpath.
To use inc-align, you should have:
- A Berkeley Aligner model which has already been trained on the relevant language combination with the following configuration settings in the
*.conf
file:forwardModels MODEL1 HMM
(inc-align supports HMM, but not Model 2)reverseModels MODEL1 HMM
mode JOINT JOINT
(inc-align assumes jointly trained models)saveParams true
alignTraining
numThreads 1
(to restrict concurrency errors)
- Data to be aligned, formatted as required by the trained model (i.e. sentence-aligned, tokenized, etc... check the Berkeley Aligner documentation)
This has only been tested with the unsupervised version of the Aligner, but it probably works for the supervised version as well. Let me know if you try!
Run the jar with the following arguments:
-p, --params PARAMS REQUIRED: Directory with trained Berkeley Aligner
-d, --data DATA REQUIRED: Directory with data to align
-f, --l2 L2 f Suffix for L2 language files
-e, --l1 L1 e Suffix for L1 language files
-h, --help
Like so:
$ java -jar inc-align-0.1.0-standalone.jar -p model-output-dir -d sentences-to-align-dir [-e l1_suffix -f l2_suffix]
In passing directory paths to the -p and -d arguments, note that tilde expansion isn't supported, but relative paths (e.g. ./
, ../../
) are.
The directory inc-align-output
will be created in the data directory, containing three files: output.{l1 suffix}
and output.{l2 suffix}
with all of the aligned sentences, and output.align
with alignments in Pharoah notation.
Attempting to load trained models sometimes throws a mysterious java.io.OptionalDataException
. In these cases, your best bet is to retrain the model, ensuring that the parameter numThreads
is set to 1
- it seems that multithread-trained models are more likely to run into these concurrency issues. However, sometimes that still doesn't work, and I'm not sure why. If you have thoughts about this, I'm very open to hearing them.
There are two standalone helper scripts:
- tknze takes a text file (ideally already sentence-aligned => one sentence per line) and tokenizes it in the style of the BA example data
- vis takes a model and two aligned data files (e.g.
data.e
anddata.f
), and writes a new filedata.vis
with their alignments visualized as distortion matrices
See comments in the files for usage.
Copyright © 2016 Kate McCurdy
Distributed under the GNU GPL v2.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.