This is a continuation of the project originally developed at Princeton, whose original files can be found this link or in the princeton branch of this repository.
See the releases page for the official releases.
This repository hosts a semantic concordance – a textual corpus (the WordNet glosses) and a lexicon (WordNet) where every content word in the corpus is linked to its sense(s) in the lexicon.
There are several corpora with WordNet sense annotations (the SemCor, the senseval datases), but only by sense-tagging the WordNet itself can we guarantee that its definitional completeness – the property that all of its definitions only use words which are already defined by WordNet.
We are annotating the corpus using this tool. It depends on the gloss corpus in the format available in this repository. Details about the format itself can be found in the tool’s repository.
cd data
split -l 1000 -a 2 annotation.json annotation-
for f in annotation-??; do mv $f $f.jl; doneNumber of tokens per kind:
for f in *.jl ; do jq -r ".tokens | .[] | .kind | .[0] " $f; done | sort | uniq -cextracting word forms:
for f in *.new; do jq -r ".tokens|.[] |.form " $f ; done | sort | uniq -c | sort -nr >> ../words.txt| sense tagged | 53212 | 0.94 |
| not sense tagged | 3350 | 0.06 |
| total taggable | 56562 | 1.00 |
among the sense tagged ones, two last lines are errors:
| 39864 | auto |
| 13348 | man |
| sense tagged | 448091 | 0.56 |
| not sense tagged | 334068 | 0.43 |
| total taggable | 792281 | 1.00 |
among the token tagged:
| 126940 | auto |
| 321151 | man |