The online demo of LiB is here: https://hub.gke2.mybinder.org/user/ray306-lib_demo-2aracalt/doc/workspaces/auto-8/tree/Quick_Demo.ipynb You can run the Jupyter notebook to see the segmentation result.
The code and data of paper Unsupervised text segmentation predicts eye fixations during reading are achieved in https://github.com/ray306/LiB-predicts-eye-fixations/
This repository achieves the code and data in paper Less is Better: A cognitively inspired unsupervised model for language segmentation
Abstract
Language users process utterances by segmenting them into many cognitive units, which vary in their sizes and linguistic levels. Although we can do such unitization/segmentation easily, its cognitive mechanism is still not clear. This paper proposes an unsupervised model, Less-is-Better (LiB), to simulate the human cognitive process with respect to language unitization/segmentation. LiB follows the principle of least effort and aims to build a lexicon which minimizes the number of unit tokens (alleviating the effort of analysis) and number of unit types (alleviating the effort of storage) at the same time on any given corpus. LiB’s workflow is inspired by empirical cognitive phenomena. The design makes the mechanism of LiB cognitively plausible and the computational requirement light-weight. The lexicon generated by LiB performs the best among different types of lexicons (e.g. ground-truth words) both from an information-theoretical view and a cognitive view, which suggests that the LiB lexicon may be a plausible proxy of the mental lexicon.
https://aclanthology.org/2020.cogalex-1.4/
Yang, J., Frank, S. L., & van den Bosch, A. (2020). Less is Better: A cognitively inspired unsupervised model for language segmentation. Proceedings of the Workshop on the Cognitive Aspects of the Lexicon, 33–45.
Open demo.ipynb by Jupyter and you could test the LiB model :)