> In the WordPiece tokenization, one word may be divided into multiple sub-word. But, how should we handle the lalel ？AM

[Maca001]

In the WordPiece tokenization, one word may be divided into multiple sub-word. But, how should we handle the lalel ？

The new part will be represented by a special label at the corresponding label location.
For example, I use a special flag ‘X’:
['Nadim', 'Ladki', 'AL-AIN', ','] -----> ['Nadim', 'Ladki', 'AL', '-', '[UNK]', ',']
['B-PER', 'I-PER', 'B-LOC', 'O'] ------> ['B-PER', 'I-PER', 'B-LOC', 'X', 'X', 'O']

Originally posted by @yuanxiaosc in google-research/bert#291 (comment)

[LawrenceLee525]

收到啦，谢谢啦

[Maca001]

ANTHONY MARCELLINUS