What is Juman++

A new morphological analyser that considers semantic plausibility of word sequences by using a recurrent neural network language model (RNNLM). Version 2 has better accuracy and greatly (>250x) improved analysis speed than the original Juman++.

Installation

System Requirements

OS: Linux, MacOS X or Windows.
Compiler: C++14 compatible (will downgrade to C++11 later)
- For, example gcc 5.1+, clang 3.4+, MSVC 2017
- We test on GCC and clang on Linux/MacOS, mingw64-gcc and MSVC2017 on Windows

CMake v3.1 or later

Building from a package

Download the package from Releases

$ tar xf jumanpp-<version>.tar.xz # decompress the package
$ cd jumanpp-<version> # move into the directory
$ mkdir bld # make a subdirectory for build
$ cd bld
$ cmake .. \
  -DCMAKE_BUILD_TYPE=Release \ # you want to do this for performance
  -DCMAKE_INSTALL_PREFIX=<prefix> # where to install Juman++
$ make install -j<parallelism>

Building from git

Generally, the differences between the package and this repository is the presence of a prebuilt model and absense of some development scripts.

$ mkdir cmake-build-dir # CMake does not support in-source builds
$ cd cmake-build-dir
$ cmake ..
$ make # -j

Usage

Quick start

% echo "魅力がたっぷりと詰まっている" | jumanpp
魅力 みりょく 魅力 名詞 6 普通名詞 1 * 0 * 0 "代表表記:魅力/みりょく カテゴリ:抽象物"
が が が 助詞 9 格助詞 1 * 0 * 0 NIL
たっぷり たっぷり たっぷり 副詞 8 * 0 * 0 * 0 "自動認識"
と と と 助詞 9 格助詞 1 * 0 * 0 NIL
詰まって つまって 詰まる 動詞 2 * 0 子音動詞ラ行 10 タ系連用テ形 14 "代表表記:詰まる/つまる ドメイン:料理・食事 自他動詞:他:詰める/つめる"
いる いる いる 接尾辞 14 動詞性接尾辞 7 母音動詞 1 基本形 2 "代表表記:いる/いる"
EOS

Main options

usage: jumanpp [options] 
  -s, --specifics              lattice format output (unsigned int [=5])
  --beam <int>                 set local beam width used in analysis (unsigned int [=5])
  -v, --version                print version
  -h, --help                   print this message
  --model <file>               specify a model location

Use --help to see more options.

Input

JUMAN++ can handle only utf-8 encoded text as an input. Lines beginning with # will be interpreted as comments.

Other

DEMO

You can play around our web demo which displays a subset of the whole lattice. The demo still uses v1 but, it will be updated to v2 soon.

Extracting diffs caused by beam configurations

You can see sentences in which two different beam configutaions produce different analyses. A src/jumandic/jpp_jumandic_pathdiff binary (source) (relative to a compilation root) does it. The only Jumandic-specific thing here is the usage of code-generated linear model inference.

Use the binary as jpp_jumandic_pathdiff <model> <input> > <output>.

Outputs would be in the partial annotation format with a full beam results being the actual tags and trimmed beam results being written as comments.

Example:

# scores: -0.602687 -1.20004
# 子がい        pos:名詞        subpos:普通名詞 <------- trimmed beam result
# S-ID:w201007-0080605751-6 COUNT:2
熊本選抜にはマリノス、アントラーズのユースに行く
        子      pos:名詞        subpos:普通名詞 <------- full beam result
        が      pos:助詞        subpos:格助詞
        い      baseform:いる   conjtype:母音動詞       pos:動詞        conjform:基本連用形
ます

Performance Notes

To get the best performance, you need to build with extended instructuion sets. If you are planning to use Juman++ only locally, specify -DCMAKE_CXX_FLAGS="-march=native".

Works best on Intel Haswell and newer processors (because of FMA and BMI instruction set extensions).

Publications and Slides

About the model itself: Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model. Hajime Morita, Daisuke Kawahara, Sadao Kurohashi. EMNLP 2015 link.
V2 Improvments: Juman++ v2: A Practical and Modern Morphological Analyzer. Arseny Tolmachev and Kurohashi Sadao. The Proceedings of the Twenty-fourth Annual Meeting of the Association for Natural Language Processing. March 2018, Okayama, Japan. (pdf, slides)
Morphological Analysis Workshop in ANLP2018 Slides: 形態素解析システムJuman++. 河原大輔, Arseny Tolmachev. (in Japanese) slides.

Authors

Arseny Tolmachev <arseny at kotonoha.ws>
Hajime Morita <hmorita at i.kyoto-u.ac.jp>
Daisuke Kawahara <dk at i.kyoto-u.ac.jp>
Sadao Kurohashi <kuro at i.kyoto-u.ac.jp>

Acknowledgement

The list of all libraries used by JUMAN++ is here.

Notice

This is a branch for the Juman++ rewrite. The original version lives in the legacy branch.

mingtao13595/jumanpp