tokenize-anything.sh on Italian

Question

Opened this issue 9 years ago · 2 comments

Hi,
I just wanted to let you know an error the tokenize-anything.sh script makes for Italian sentences, that is it doesn't split "C'è" ("There's").

Answer 1 · 2015-07-25T22:41:17.000Z

This also applies to other contractions whose second part is "'è".

Answer 2 · 2015-07-25T23:19:13.000Z

Examples of other contractions that should be split, but aren't:

l'uomo
all'interno
nell'obligo

These involve articles. Before a vowel, definite articles are spelled l'. Combining with prepositions yields all', dall', dell', nell', sull'. The feminine indefinite article is realized as un' before a vowel.