/bag-of-substring-embedder

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Bag-of-Substring Word Embedder

This is the code used in Generalizing Word Embeddings using Bag of Subwords (EMNLP 2018).

Dependencies

  • Python 3.5
  • gensim 3.1 for loading word2vec vectors.
  • tqdm 4.15 for progress reporting.
  • fastText for word similarity evaluation script.
  • Mimick for sentense-level bidirectional LSTM model.

The last two are included as submodules. Initialize the submodules using

git submodule init
git submodule update

Contents

  • bos-demo.sh: a script showcasing the usage.
  • bos-pred.py: generate vectors for word queries.
  • bos-train.py: train a BoS model.
  • bos.py: BoS model definition.
  • data-processing.py: prepare target word vectors.

Usage

To start a demo run, simply ./bos-demo.sh.

To preform word similarity evluation, refer to fastText/eval.py.

To perform POS and morphosyntactic tagging evaluation, refer to Mimick/model.py.