shilad/wikibrain

Wikification produces incorrect text locations

Yuropa opened this issue · 1 comments

Special characters (like commas, colons, semicolons, etc.) and whitespace besides a single space (newlines, 2 or more spaces, etc.) are ignored when calculating the LocalLink location in a piece of wikified text.

I have only tested this with WebSailWikifier, but it may affect other wikifiers. I suspect it has to do with how the input text is tokenized.

Never mind... Turns out I had an old piece of code which did this trimming and I had forgot to remove it :)