===========================
====== README =======
===========================
This package contains the code underlying the SCAN model as described in
Frermann and Lapata (2016). The code can be used to (a) create binary files
containing time-stamped input documents (the required input to SCAN); and (b)
to train scan models.
This bundle contains three parts
(1) main.zip
the main functions. Run code from this directory. Also contains example
input and output
(2) scan_lib.zip
the core code underlying SCAN (my code)
(3) other_lib.zip
libraries that are used but not part of the core go libraries. Each of these must
be placed in the same directory as scan_lib.zip
To run the code, navigate into your 'main' directory and run
EITHER the pre-compiled binary (should work out of the box)
./dynamic-senses -parameter_file=/path/to/parameters/ [-create_corpus] -store={true,false}
OR the code itself (requires to have go installed)
go run *.go -parameter_file=/path/to/parameters/ [-create_corpus] -store={true,false}
The command-line parameters are:
- parameter_file
path to a text file containing all parameters (see below)
-create_corpus
an optional parameter. If it is set a binary corpus will be created. Otherwise a model will be trained
-store
indicates whether target word-specific corpora should be stored
To run the code itself (second option above i.e., not just the binary) golang (https://golang.org/doc/)
must be installed. I use version go1.7.4 (might be safest to use the same). All required
packages to run this code should be part of this bundle.
----------------------------------
----- running out of the box -----
----------------------------------
I include an example for all necessary test files:
- a parameters.txt (which has my own hard-coded paths, MUST BE CHANGED)
- corpus file under main/test_input/corpus.txt
- a file with target words under main/test_input/targets.txt
Change the paths in the parameters.txt to your own paths (to the corpus / targets fies),
and run
(b) to create a binary corpus
./dynamic_senses -parameter_file=path/to/parameters.txt -create_corpus -store=true
(a) to train models
./dynamic_senses -parameter_file=path/to/parameters.txt -store=true
---------------------------
--- the parameters file ---
---------------------------
All parameters are specified in a parameters which must be passed as input to
the program. Best see the included 'parameters.txt' examples file. This includes
- paths to underlying text corpora, target word sets, etc
- model parameters (see paper for explanation)
- sampler parameters (number of iterations)
- parameters regarding the time start/end/intervals of interest
- parameters to optionally restrict the minimum number of available documents (to ignore highly
infrequent words) and / or maximum number of available documents per time interval
(to get managable-size corpora for very high-frequent words)
------------------------------
Creating binary input corpora
------------------------------
Takes a text file and a list of target words, and a 'document length' specification. Outputs
a binary corpus with target-word specific, time-stamped documents. Document length refers to
the size of the context window considered around the target word, e.g., 5 words.
[The corpus has words mapped to unique ID identifiers, and contains dictionaries mapping from
word strings to IDs and back]
It takes the following parameters (all specified in parameters.txt)
- text_corpus
path to a text file which in each line contains a number indicating
a year (of origin) followed by a \tab\ character followed by the corresponding text
from that year of origin. The same year can be listed multiple times:
YEAR \tab\ text ....
YEAR \tab\ text ....
....
- target_words
path to a text file containing all target words of interest
whose meaning should be tracked, one word per line.
- window_size
It also requires a specification of the window size (i.e, the context window to consider, as
explained above)
- bin_corpus_store
path to the location the binary corpus should be stored
------------------------------
Training a SCAN model
------------------------------
Once we have a binary corpus of time-stamped documents as explained above full_corpus_path, we can train SCAN
models that track meaning change of individual target words. To do this we
(1) extract a target word-specific corpus from the underlying binary corpus. It contains only time-tagged
documents with the specified target word. It converts the absolute times in the underlying corpus (e.g, 1764, 1993, ...)
to time intervals (0, 1, ..., T) based on the start_time, end_time and time_interval parameters (see below).
It takes the following parameters (all specified in parameters.txt):
- full_corpus_path
path to the underlying binary corpus
- start_time
the earliest time stamp in the underlying corpus to be considered
- end_time
the latest time stamp in the underlying corpus to be considered
- time_interval
the length of time intervals into which the span [start_time, end_time] is to be split
e.g., if start_time=1700 , end_time=2000 , time_interval=10 then documents are binned into 10-year bins
and all documents from before 1700 and after 2000 are ignored. Documents from 1700-1709 are assigned to
bin 0, documents from 1710-1719 are assigned to bin 1 and so on.
- word_corpus_path
path to a location where word-specific corpora are stored.
The filename reflects the choice of start_time / end_time / time_interval,
e.g., corpus_s1700_e2009_i10.bin for the example above
(2) pass this corpus to the model and train the model with MCMC inference. It creates a model and human-readable output:
- model.bin the trained model binary and
- output.dat human-readable model output, namely for each time slice its distribution over senses,
and for each sense in each time slice, it's distribution over words (as the set of most
highly associated words).
It takes the following parameters (all specified in parameters.txt)
- output_path
directory in which files output.dat and model.bin are to be stored
If the output directory and files (a) and (b) already exist (from a previous run) it moves the old files
to output_old.dat and model_old.bin
- kappaF, kappaK, a0, b0, num_top
model parameters; check paper (or ask me!) for explanations
- iterations
number of training iterations
- min_doc_per_word
the model doesn't work well if there are very few documents for a target word available. You may want to
only learn models per target words that occur at least N (~ 500?) times in the data
- max_docs_per_slice
some words occur extremely often. To get a managable size input corpus you can restrict the number of documents
to consider per time interval
-------------------------------
Understanding the output
-------------------------------
The program creates a human-readable output file for each target word in [outptut_path]/word/output.dat
The included directories under main/test_input/output/ contain output for models trained on the corpus in
main/test_input/corpus.txt for the target words in main/test_input/targets.txt
Models were trained to learn
-- from the corpus.txt file (containing some language from between 1700 and 2009)
-- K=8 senses per word
-- Start_time=1700, end_time=2009, intervals=20 --> obtaining 16 20-year time intervals in total
In each output.dat file:
-- K indexes the sense ID
-- T indexes the time slice ID
-- After each sense / time: The top 10 words with highest probability under each sense are listed
-- The bottom line of each block (p(w|s)....) sums over all senses, i.e., shows the most highly associated words
for a particular time, ignoring sense information.
The file output.dat contains the same information in two ways.
*** per type***
First, I list *by sense* the representation of the
same sense (k=0...K) for each time slice. The first number indicates the sense's prevalence at that par-
ticular time (as a probability, between 0 and 1). Look for example in main/example_input/output/power/output.dat.
We can see that sense 2 (the block with K=2) seems to relate to the 'electricity' sense of power (it has
a highly associated words 'battery', 'plant', etc especially towards later times, and tts prevalence increases
towards later time slices.
*** per time ***
This lists the senses associated with each time interval (the content of the lines is the same as above). E.g.,
the times associated with T=0 (first block under 'per_time') shows that senses K=7 has high probablility
and sense K=2 (the 'power' sense including e.g., the word 'dynamo') has low probability. Senses K=2 refers to
the 'mental' power.
Output for the targets 'battery' and 'transport' is also included.