/TextDiversity

Primary LanguagePython

TextDiversity

This code is designed to measure various kinds of linguistic diversity using similarity-sensitive hill numbers (SSHN). Originally adapted from the study of species diversity in ecology, SSHNs are used to characterize the effective number of species in a population. Within the context of NLP, species are the linguistic units of interest (e.g. words, parse trees, etc) and the population is a corpus of documents. For example, if the "token semantic diversity" of a corpus is 9, this can be read as the corpus containing 9 semantic concepts.

Example Usage

import pandas as pd
from textdiversity import (
    TokenSemantics, DocumentSemantics, AMR, # semantics
    DependencyParse, ConstituencyParse,     # syntactical
    PartOfSpeechSequence,                   # morphological
    Rhythmic                                # phonological
)

corpus1 = ['one massive earth', 'an enormous globe', 'the colossal world'] # unique words
corpus2 = ['basic human right', 'you were right', 'make a right']          # lower unigram diversity

metrics = [
    TokenSemantics(), DocumentSemantics(), AMR(), 
    DependencyParse(), ConstituencyParse(), 
    PartOfSpeechSequence(), 
    Rhythmic()
]

results = []
for metric in metrics:
    results.append({
        "metric": metric.__class__.__name__,
        "corpus1": metric(corpus1),
        "coprus2": metric(corpus2)
    })

df = pd.DataFrame(results)
metric corpus1 coprus2
0 TokenSemantics 7.42473 7.99136
1 DocumentSemantics 1.18850 1.65927
2 AMR 2.76379 1.71087
3 DependencyParse 1.00000 1.88204
4 ConstituencyParse 1.00001 1.88989
5 PartOfSpeechSequence 1.17621 1.80000
6 Rhythmic 1.36364 1.81230

Installation

pip install textdiversity

Some of the text diversity measures rely on software that must be installed separately. For your particular OS, follow the instructions below:

Windows

  1. Phoneme Diversity
    • Phonemizer
      • Visit: https://github.com/espeak-ng/espeak-ng/releases
      • Install either espeak-ng-X64.msi or espeak-ng-X86.msi (This project used Release 1.51)
      • Add Phonemizer environement variable ([reference])
        • Navigate: Windows Key > Edit System Environment Variables > Environment Variables
        • Add New Variable:
          • Variable Name: PHONEMIZER_ESPEAK_LIBRARY
          • Variable Value: C:\Program Files\eSpeak NG\libespeak-ng.dll
  2. Syntactical Diversity (non-core functionality)
    • Dependency Parse Tree Visualization
      • UNKOWN SETUP STEPS

Linux

  1. Phoneme Diversity
    • Phonemizer
      • sudo apt-get install espeak-ng
  2. Syntactical Diversity
    • Dependency Parse Tree Visualization (non-core functionality)
      • sudo apt-get install python3-dev graphviz libgraphviz-dev pkg-config