sentence segmentation error

[jmugan]

I know that sentence segmentation is more difficult than it appears, but this looks like an error that spaCy might want to know about. Or if there is something I am doing wrong, it would be good for me to know :)

text = "He has a Ph.D. in tacology."
tokens = parser(text)
for sent in tokens.sents:
    print("Sentence:")
    print(sent)
    print()

results in

Sentence:
He has a Ph.

Sentence:
D. in tacology.

In Python 3.5, spaCy version 1.1.2.

[honnibal]

Thanks!

One of the features of 1.0 I haven't documented yet is that it's now easy to fix this sort of thing at run-time, without messing with data files.

You should be able to do:

import spacy.en

spacy.en.English.Defaults.tokenizer_exceptions["Ph.D."] = [{"F": "Ph.D."}]

What you're doing here is telling the tokenizer, "when you see this chunk, 'Ph.D.', process it into these tokens". The list you're giving it here has just one token, and you've specified its form (with the F key). You can also specify its POS and lemma (with P and L). This is a little dinky --- it should support full token attributes.

[jmugan]

Cool! That works!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.