/LSCDetection

Data Sets and Models for Evaluation of Lexical Semantic Change Detection

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

LSCDetection

General

Data Sets and Models for Evaluation of Lexical Semantic Change Detection.

If you use this software for academic research, please cite this paper:

Also make sure you give appropriate credit to the below-mentioned software this repository depends on.

Parts of the code rely on DISSECT, gensim, numpy, scikit-learn, scipy, VecMap.

Usage

The scripts should be run directly from the main directory. If you wish to do otherwise, you may have to change the path you add to the path attribute in sys.path.append('./modules/') in the scripts. All scripts can be run directly from the command line:

python3 representations/count.py <corpDir> <outPath> <windowSize>

e.g.

python3 representations/count.py corpora/test/corpus1/ test_matrix1 1

The usage of each script can be understood by running it with help option -h, e.g.:

python3 representations/count.py -h

We recommend you to run the scripts within a virtual environment with Python 3.7.4. Install the required packages running pip install -r requirements.txt. (See also error sources.)

Models

A standard model of LSC detection executes three consecutive steps:

  1. learn semantic representations from corpora (representations/)
  2. align representations (alignment/)
  3. measure change (measures/)

As an example, consider a very simple model (CNT+CI+CD) going through these steps:

  1. learn count vectors from each corpus to compare (representations/count.py)
  2. align them by intersecting their columns (alignment/ci_align.py)
  3. measure change with cosine distance (measures/cd.py)

You can apply this model to the testing data using the following commands:

    python3 representations/count.py corpora/test/corpus1/ test_matrix1 1
    python3 representations/count.py corpora/test/corpus2/ test_matrix2 1

    python3 alignment/ci_align.py test_matrix1 test_matrix2 test_matrix1_aligned test_matrix2_aligned

    python3 measures/cd.py -s testsets/test/targets.tsv test_matrix1_aligned test_matrix2_aligned test_results.tsv

Input Format: All the scripts in this repository can handle two types of matrix input formats:

To learn more about how matrices are loaded and stored check out modules/utils_.py.

The scripts assume a corpus format of one sentence per line in UTF-8 encoded (optionally zipped) text files. You can specify either a file path or a folder. In the latter case the scripts will iterate over all files in the folder.

Semantic Representations

Name Code Type Comment
Count representations/count.py VSM
PPMI representations/ppmi.py VSM
SVD representations/svd.py VSM
RI representations/ri.py VSM
SGNS representations/sgns.py VSM
SCAN repository TPM - different corpus input format

Table: VSM=Vector Space Model, TPM=Topic Model

Alignment

Name Code Applicability Comment
CI alignment/ci_align.py Count, PPMI
SRV alignment/srv_align.py RI - consider using more powerful TRIPY
OP alignment/map_embeddings.py SVD, RI, SGNS - drawn from VecMap
- for OP- and OP+ see scripts/
VI alignment/sgns_vi.py SGNS - bug fixes 27/12/19 (see script for details)
WI alignment/wi.py Count, PPMI, SVD, RI, SGNS - consider using the more advanced Temporal Referencing

Measures

Name Code Applicability Comment
CD measures/cd.py Count, PPMI, SVD, RI, SGNS
LND measures/lnd.py Count, PPMI, SVD, RI, SGNS
JSD - SCAN
FD measures/freq.py from corpus - log-transform with measures/trsf.py
- get difference with measures/diff.py
TD measures/typs.py Count as above
HD measures/entropy.py Count as above

Parameter Settings

Find detailed notes on model performances and optimal parameter settings in these papers.

Evaluation

The evaluation framework of this repository is based on the comparison of a set of target words across two corpora. Hence, models can be evaluated on a triple (dataset, corpus1, corpus2), where the dataset provides gold values for the change of target words between corpus1 and corpus2.

Datasets

Dataset Language Corpus 1 Corpus 2 Download Comment
DURel German DTA18 DTA19 Dataset, Corpora - version from Schlechtweg et al. (2019) at testsets/durel/
SURel German SDEWAC COOK Dataset, Corpora - version from Schlechtweg et al. (2019) at testsets/surel/
SemCor LSC English SEMCOR1 SEMCOR2 Dataset, Corpora
SemEval Eng English CCOHA 1810-1860 CCOHA 1960-2010 Dataset, Corpora
SemEval Ger German DTA 1800-1899 BZND 1946-1990 Dataset, Corpora
SemEval Lat Latin LatinISE -200-0 LatinISE 0-2000 Dataset, Corpora
SemEval Swe Swedish Kubhist2 1790-1830 Kubhist2 1895-1903 Dataset, Corpora
RuSemShift1 Russian RNC 1682-1916 RNC 1918-1990 Dataset
RuSemShift2 Russian RNC 1918-1990 RNC 1991-2016 Dataset

We provide several evaluation pipelines, downloading the corpora and evaluating the models on (most of) the above-mentioned datasets, see pipelines.

Metrics

Name Code Applicability Comment
Spearman correlation evaluation/spr.py DURel, SURel, SemCor LSC, SemEval* - outputs rho (column 3) and p-value (column 4)
Average Precision evaluation/ap.py SemCor LSC, SemEval* - outputs AP (column 3) and random baseline (column 4)

Consider uploading your results for DURel as a submission to the shared task Lexical Semantic Change Detection in German and for SemEval* to SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection .

Pipelines

Under scripts/ you find an example of a full evaluation pipeline for the models on two small test corpora. Assuming you are working on a UNIX-based system, first make the scripts executable with

chmod 755 scripts/*.sh

Then run

bash -e scripts/run_test.sh

The script first reads the two gzipped test corpora corpora/test/corpus1/ and corpora/test/corpus2/. Then it produces model predictions for the targets in testsets/test/targets.tsv and writes them under results/. It finally writes the Spearman correlation between each model's predictions and the gold rank (testsets/test/gold.tsv) under the respective folder in results/. Note that the gold values for the test data are meaningless, as they were randomly assigned.

We also provide a script for each dataset running all the models on it including necessary downloads. For this run either of

bash -e scripts/run_durel.sh
bash -e scripts/run_surel.sh
bash -e scripts/run_semcor.sh
bash -e scripts/run_semeval*.sh

You may want to change the parameters in scripts/parameters_durel.sh, etc. (e.g. vector dimensionality, iterations), as running the scripts on the full parameter set may take several days and require a large amount of disk space.

Important Changes

  • September 1, 2019: Python scripts were updated from Python 2 to Python 3.
  • December 27, 2019: bug fixes in alignment/sgns_vi.py (see script for details)
  • March 23, 2020: updates in representations/ri.py and alignment/srv_align.py (see scripts for details)

Error Sources

  • if you are on a Windows system and get error messages like [bash] $'\r': command not found, consider removing trailing '\r' characters with sed -i 's/\r$//' scripts/*.sh

BibTex

@inproceedings{Schlechtwegetal19,
	title = {{A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains}},
	author = {Dominik Schlechtweg and Anna H\"{a}tty and Marco del Tredici and Sabine {Schulte im Walde}},
    booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
	year = {2019},
	address = {Florence, Italy},
	publisher = {Association for Computational Linguistics},
	pages = {732--746},
    doi = {10.18653/v1/P19-1072}
}

@inproceedings{SchlechtwegWalde20,
	title = {{Simulating Lexical Semantic Change from Sense-Annotated Data}},
	author = {Dominik Schlechtweg and Sabine {Schulte im Walde}},
	year = {2020},
	booktitle = {{The Evolution of Language: Proceedings of the 13th International Conference (EvoLang13)}},
	editor = {Ravignani, A. and Barbieri, C. and Martins, M. and Flaherty, M. and Jadoul, Y. and Lattenkamp, E. and Little, H. and Mudd, K. and Verhoef, T.},
	url = {http://brussels.evolang.org/proceedings/paper.html?nr=9},
	doi = {10.17617/2.3190925}
}