PyDSM is a lightweight Python 3 framework for building and exploring distributional semantic models with focus on extensibility and ease of use. While mostly developed as a personal project, I hope it could still be found useful for others.
Building a DSM with PyDSM is easy:
In [1]: import pydsm, plainstream
In [2]: wikitext = plainstream.get_text(language='en', max_words=1000, tokenize=True)
In [3]: cooc = pydsm.build(pydsm.CooccurrenceDSM, window_size=(2,2), corpus=wikitext, lower_threshold=3)
Building collocation matrix from corpus....
Total time: 1.75 s
In [4]: cooc
Out[4]:
CooccurrenceDSM
Vocab size: 445
[61, 61] a been the ...
that 0.0 0.0 2.0 ...
some 0.0 0.0 0.0 ...
'' 1.0 0.0 2.0 ...
... ... ... ... ...
Please see the tutorial for a quick introduction of the package.
- Build distributional semantic models from text corpora (Cooccurrence matrix and Random Indexing models included).
- Find nearest neighbors using common similarity measures.
- Apply common weighting techniques, such as positive pointwise mutual information.
- Simple DSM visualizations.
Download the package, and type:
$ python setup.py install
The package is only tested on python 3.4.
Please make sure you have Cython installed.
A lot of inspiration comes from the DISSECT toolkit, a part of the COMPOSES project. Many headaches were avoided from inspecting their work.