LastRyan opened this issue 4 years ago · 0 comments
if len(words_to_tokens) + len(input_ids) + 1 > self.config.max_seq_length: to if len(word_tokens) + len(input_ids) + 1 > self.config.max_seq_length: ?
if len(words_to_tokens) + len(input_ids) + 1 > self.config.max_seq_length:
if len(word_tokens) + len(input_ids) + 1 > self.config.max_seq_length: