UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte (while running build_openwebtext_pretraining_dataset.py )

[elyorman]

electra_p3) ubuntu@nipa2020-0706:~/EL/electra/electra$ python3 build_openwebtext_pretraining_dataset.py --data-dir DATA_DIR --num-processes 5 Job 0: Creating example writer Job 1: Creating example writer Process Process-1: Job 2: Creating example writer Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "build_openwebtext_pretraining_dataset.py", line 47, in write_examples do_lower_case=args.do_lower_case File "/home/ubuntu/EL/electra/electra/build_pretraining_dataset.py", line 126, in __init__ do_lower_case=do_lower_case) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 116, in __init__ self.vocab = load_vocab(vocab_file) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 78, in load_vocab token = convert_to_unicode(reader.readline()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 179, in readline return self._prepare_value(self._read_buf.ReadLineAsString()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 98, in _prepare_value return compat.as_str_any(val) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 123, in as_str_any return as_str(value) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 93, in as_text return bytes_or_text.decode(encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte Process Process-2: Job 3: Creating example writer Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "build_openwebtext_pretraining_dataset.py", line 47, in write_examples do_lower_case=args.do_lower_case File "/home/ubuntu/EL/electra/electra/build_pretraining_dataset.py", line 126, in __init__ do_lower_case=do_lower_case) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 116, in __init__ self.vocab = load_vocab(vocab_file) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 78, in load_vocab token = convert_to_unicode(reader.readline()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 179, in readline return self._prepare_value(self._read_buf.ReadLineAsString()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 98, in _prepare_value return compat.as_str_any(val) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 123, in as_str_any return as_str(value) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 93, in as_text return bytes_or_text.decode(encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte Process Process-3: Job 4: Creating example writer Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "build_openwebtext_pretraining_dataset.py", line 47, in write_examples do_lower_case=args.do_lower_case File "/home/ubuntu/EL/electra/electra/build_pretraining_dataset.py", line 126, in __init__ do_lower_case=do_lower_case) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 116, in __init__ self.vocab = load_vocab(vocab_file) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 78, in load_vocab token = convert_to_unicode(reader.readline()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 179, in readline return self._prepare_value(self._read_buf.ReadLineAsString()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 98, in _prepare_value return compat.as_str_any(val) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 123, in as_str_any return as_str(value) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 93, in as_text return bytes_or_text.decode(encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte Process Process-4: Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "build_openwebtext_pretraining_dataset.py", line 47, in write_examples do_lower_case=args.do_lower_case File "/home/ubuntu/EL/electra/electra/build_pretraining_dataset.py", line 126, in __init__ do_lower_case=do_lower_case) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 116, in __init__ self.vocab = load_vocab(vocab_file) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 78, in load_vocab token = convert_to_unicode(reader.readline()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 179, in readline return self._prepare_value(self._read_buf.ReadLineAsString()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 98, in _prepare_value return compat.as_str_any(val) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 123, in as_str_any return as_str(value) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 93, in as_text return bytes_or_text.decode(encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte Process Process-5: Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "build_openwebtext_pretraining_dataset.py", line 47, in write_examples do_lower_case=args.do_lower_case File "/home/ubuntu/EL/electra/electra/build_pretraining_dataset.py", line 126, in __init__ do_lower_case=do_lower_case) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 116, in __init__ self.vocab = load_vocab(vocab_file) File "/home/ubuntu/EL/electra/electra/model/tokenization.py", line 78, in load_vocab token = convert_to_unicode(reader.readline()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 179, in readline return self._prepare_value(self._read_buf.ReadLineAsString()) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/lib/io/file_io.py", line 98, in _prepare_value return compat.as_str_any(val) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 123, in as_str_any return as_str(value) File "/home/ubuntu/anaconda3/envs/electra_p3/lib/python3.7/site-packages/tensorflow_core/python/util/compat.py", line 93, in as_text return bytes_or_text.decode(encoding) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte
I am facing this error while pretraining data