atilika/kuromoji

Question: how to obtain multiple parsings?

fasiha opened this issue · 5 comments

fasiha commented

MeCab has a -N flag with which a user can specify the top-N results to get back. On http://www.atilika.org/ the Viterbi algorithm's output graph shows all possible morphemes, along with the cost of each path, so I'm sure it's possible to get the top, say, five results, but is there a simpler way to get this, the equivalent of mecab -N 5? I'm using UniDic. Thank you 🙇!

fasiha commented

I see that, since I asked this, multiTokenizeNBest has been added to TokenizerBasevia d0ed0eb by EmanuelGedin! Awesome!

Question: I’ve been using Maven, e.g., for kuromoji-unidic but these were last updated in September 2015. I’ll try to get my build tool (leiningen) to use a cloned Git repo but I was wondering if there were plans on bumping the versions on Maven soon?

Thanks!

cmoen commented

We're planning on publishing a new version to Maven soon. We'd like to let the n-best APIs to bake a little bit before we release the new version, though. Any feedback you have on usage, etc. would be greatly appreciated. Thanks!

  • 😄1
fasiha commented

Thanks for the info!

This is probably not the best place to ask about it, forgive me, but:

  1. 何できた？, vs.
  2. バスできた。

With the first sentence, multiTokenizeNBest with UniDic returns several tokenizations 😄, but with the second, it returns an empty list 😭. I’m testing with 1fad6cc (HEAD as of yesterday), and in my informal testing, there’re a number of other sentences for which multiTokenizeNBest returns the empty list.

Any suggestions?

(In both these sentences, the lowest-cost tokenization uses 出来る, instead of で＋来る, which is what I expected. For the first sentence above, multiTokenizeNBest includes this expected tokenization as in the top-3 list.)

Update: same problem happens with IPADIC too, so it’s not a UniDic issue.

Update the second: multiTokenize with a high, but not too high, costSlack does work! multiTokenize with costSlack = 214748364 (one-tenth of Integer.MAX_VALUE) works as expected. Somewhere between 214'748'364 and 2'147'483'647=Integer.MAX_VALUE, something bad happens.

cmoen commented

Sorry for the slow response here. Emanuel is looking into a fix and we hope to have something you can test soon.

  • 👍1
emmanuellegedin commented

I found the overflow error. A fix will be coming soon!

  • 👍1
  • 😄1