buildscript for Kuromoji with mecab-neologd
These scripts to build a Lucene Kuromoji or Atilika Kuromoji with bundled mecab-ipadic-NEologd.
What's Lucene Kuromoji
Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java.
Kuromoji is morphological analyzer which is included in Apache Lucene.
What's Atilika Kuromoji
Kuromoji is an open source Japanese morphological analyzer written in Java.
What's NEologd
mecab-ipadic-NEologd : Neologism dictionary for MeCab
Note: These build scripts are supporting is only IPA dictionary.
Supported versions
Lucene Kuromoji: 4.x, 5.x, 6.x
Atilika Kuromoji: 0.9.0
Usage
Requirements
To use this script, you must install the following software.
- JDK
- Apache Ant (for Lucene Kuromoji)
- Apache Maven (for Atilika Kuromoji)
- Git
- make
- curl
- iconv
- xz
- Perl
- MeCab (mecab/mecab-config) (optional, auto install)
- C++ Compiler (when MeCab is installed automatically)
Note: Many CPU and memory resource are used by a build. About 5-6 GB of JavaVM heap is needed at present.
Build Lucene Kuromoji for mecab-ipadic-NEologd
Install
$ git clone https://github.com/kazuhira-r/kuromoji-with-mecab-neologd-buildscript
or
$ wget https://raw.githubusercontent.com/kazuhira-r/kuromoji-with-mecab-neologd-buildscript/master/build-lucene-kuromoji-with-mecab-ipadic-neologd.sh
Please to grant execute permissions.
$ chmod a+x build-lucene-kuromoji-with-mecab-ipadic-neologd.sh
Build
In any directory, please run the script.
$ /path/to/build-lucene-kuromoji-with-mecab-ipadic-neologd.sh
The setting when execute, is indicated.
### [2016-12-18 17:57:02] [main] [INFO] START.
####################################################################
applied build options.
[Auto Install MeCab Version ] ... mecab-0.996
[mecab-ipadic-NEologd Tag (-N)] ... master
[install adjective ext (-T)] ... 0
[Max BaseForm Length ] ... 15
[Lucene Version Tag (-L)] ... releases/lucene-solr/6.3.0
[Kuromoji build Max Heapsize (-M)] ... 6g
[Kuromoji JAR File Output Directory Name (-o)] ... .
[Kuromoji Package Name (-p)] ... org.apache.lucene.analysis.ja
####################################################################
That were built JAR file will be created in user specified directory (default: current directory) where you run the script.
$ ls -l
total 51832
-rw-rw-r-- 1 xyz xyz 51655324 Dec 18 18:05 lucene-analyzers-kuromoji-ipadic-neologd-6.3.0-20161215.jar
drwxrwxr-x 6 xyz xyz 4096 Dec 18 18:02 lucene-solr
drwxrwxr-x 8 xyz xyz 4096 Jul 23 00:32 mecab
drwxr-xr-x 8 xyz xyz 4096 Jul 23 00:31 mecab-0.996
-rw-rw-r-- 1 xyz xyz 1398663 Jul 23 00:31 mecab-0.996.tar.gz
drwxrwxr-x 9 xyz xyz 4096 Dec 18 17:59 mecab-ipadic-neologd
In this case, it is "lucene-analyzers-kuromoji-ipadic-neologd-6.3.0-20161215.jar" JAR file that was built.
JAR file naming
Naming of a JAR file of a build result is as follows.
naming:
lucene-analyzers-kuromoji-ipadic-neologd-[Lucene Version]-[mecab-ipadic-NEologd dictionary date].jar
example:
lucene-analyzers-kuromoji-ipadic-neologd-6.3.0-20161215.jar
Build options
- -N - branch or tag name in mecab-ipadic-NEologd, included in a build. default: master
- -T - install adjective ext. if you want enable, specified 1. default: 0
- -L - branch or tag name in Apache Lucene of a build target. default: current Apache Lucene latest release tag.
- -M - Kuromoji build max heapsize.
- -o - generated Kuromoji JAR file output directory. (default: . (current directory))
- -p - package name at the time of a build. default: org.apache.lucene.analysis.ja (original package)
Build Atilika Kuromoji for mecab-ipadic-NEologd
Install
$ git clone https://github.com/kazuhira-r/kuromoji-with-mecab-neologd-buildscript
or
$ wget https://raw.githubusercontent.com/kazuhira-r/kuromoji-with-mecab-neologd-buildscript/master/build-atilika-kuromoji-with-mecab-ipadic-neologd.sh
Please to grant execute permissions.
$ chmod a+x build-atilika-kuromoji-with-mecab-ipadic-neologd.sh
Build
In any directory, please run the script.
$ /path/to/build-atilika-kuromoji-with-mecab-ipadic-neologd.sh
The setting when execute, is indicated.
### [2016-12-18 23:10:54] [main] [INFO] START.
####################################################################
applied build options.
[Auto Install MeCab Version ] ... mecab-0.996
[mecab-ipadic-NEologd Tag (-N)] ... master
[install adjective ext (-T)] ... 0
[Kuromoji Version Tag (-K)] ... 0.9.0
[Kuromoji build Max Heapsize (-M)] ... 7g
[Kuromoji JAR File Output Directory Name (-o)] ... .
[Kuromoji Package Name (-p)] ... com.atilika.kuromoji.ipadic
####################################################################
That were built JAR file will be created in user specified directory (default: current directory) where you run the script.
$ ls -l
total 133572
drwxrwxr-x 10 xyz xyz 4096 Dec 18 23:13 kuromoji
-rw-rw-r-- 1 xyz xyz 135352388 Dec 18 23:33 kuromoji-ipadic-neologd-0.9.0-20161215.jar
drwxrwxr-x 8 xyz xyz 4096 Dec 18 22:39 mecab
drwxr-xr-x 8 xyz xyz 4096 Dec 18 22:39 mecab-0.996
-rw-rw-r-- 1 xyz xyz 1398663 Jul 23 00:32 mecab-0.996.tar.gz
drwxrwxr-x 9 xyz xyz 4096 Dec 18 23:11 mecab-ipadic-neologd
In this case, it is "kuromoji-ipadic-neologd-0.9.0-20161215.jar" JAR file that was built.
JAR file naming
Naming of a JAR file of a build result is as follows.
naming:
kuromoji-ipadic-neologd-[Atilika Kuromoji Version]-[mecab-ipadic-NEologd dictionary date].jar
example:
kuromoji-ipadic-neologd-0.9.0-20161215.jar
Build options
- -N - branch or tag name in mecab-ipadic-NEologd, included in a build. default: master
- -T - install adjective ext. if you want enable, specified 1. default: 0
- -K - branch or tag name in Atilika Kuromoji of a build target. default: current Atilika Kuromoji latest release tag.
- -M - Kuromoji build max heapsize.
- -o - generated Kuromoji JAR file output directory. (default: . (current directory))
- -p - package name at the time of a build. default: com.atilika.kuromoji.ipadic (original package)
Internal Process
This script, perform the following processing.
- Check the installation of MeCab, Installing MeCab in the current directory unless MeCab is not installed
- Clone mecab-ipadic-NEologd
- Generate a dictionary CSV(using libexec/make-mecab-ipadic-neologd.sh -L)
- Clone Apache Lucene or Atilika Kuromoji source code
- (Lucene Kuromoji only) Edit Apache Lucene Kuromoji's build.xml
- Rename package name, when being necessary
- Build Kuromoji and dictionary with mecab-ipadic-NEologd
- Copy JAR file to specified directory (default: current directory)
LICENSE
Copyright © 2015, 2016 kazuhira-r
Licensed under the Apache License, Version 2.0