Cannot Finetune 11B on Cloud VM
jiachangliu opened this issue · 7 comments
jiachangliu commented
Here is how I try to finetune the 11B model inside the VM:
pip install t5[gcp]
export PROJECT=my-project-xxx
export ZONE=us-central1-b
export BUCKET=gs://my-path-xxx/
export TPU_NAME=t5-tpu
export TPU_SIZE=v3-8
export DATA_DIR="${BUCKET}/data"
export MODEL_DIR="${BUCKET}/models/11B/SST2"
ctpu up --name=$TPU_NAME --project=$PROJECT --zone=$ZONE --tpu-size=$TPU_SIZE --tpu-only --noconf
t5_mesh_transformer \
--tpu="${TPU_NAME}" \
--gcp_project="${PROJECT}" \
--tpu_zone="${ZONE}" \
--model_dir="${MODEL_DIR}" \
--t5_tfds_data_dir="${DATA_DIR}" \
--gin_param="utils.tpu_mesh_shape.model_parallelism = 4" \
--gin_param="utils.tpu_mesh_shape.tpu_topology = '${TPU_SIZE}'" \
--gin_param="utils.run.train_dataset_fn = @t5.models.mesh_transformer.tsv_dataset_fn" \
--gin_param="tsv_dataset_fn.filename = 'gs://my-path-to-data-xxx/SST-2-train.tsv'" \
--gin_file="gs://t5-data/pretrained_models/11B/operative_config.gin" \
--gin_param="utils.run.save_checkpoints_steps=1000" \
--gin_file="learning_rate_schedules/constant_0_001.gin" \
--gin_param="run.train_steps = 1001000" \
--gin_param="utils.run.batch_size=('tokens_per_batch', 1024)"
I received the following error message:
W0908 15:45:33.807444 140372839036736 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
File "/opt/conda/bin/t5_mesh_transformer", line 8, in <module>
sys.exit(console_entry_point())
File "/opt/conda/lib/python3.7/site-packages/t5/models/mesh_transformer_main.py", line 254, in console_entry_point
app.run(main)
File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/opt/conda/lib/python3.7/site-packages/t5/models/mesh_transformer_main.py", line 248, in main
model_dir=FLAGS.model_dir)
File "/opt/conda/lib/python3.7/site-packages/gin/config.py", line 1069, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/opt/conda/lib/python3.7/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.__traceback__) from None
File "/opt/conda/lib/python3.7/site-packages/gin/config.py", line 1046, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/opt/conda/lib/python3.7/site-packages/mesh_tensorflow/transformer/utils.py", line 2602, in run
checkpoint_input_pipeline=checkpoint_input_pipeline)
File "/opt/conda/lib/python3.7/site-packages/mesh_tensorflow/transformer/utils.py", line 1815, in train_model
estimator.train(input_fn=input_fn, max_steps=train_steps, hooks=hooks)
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3102, in train
rendezvous.raise_errors()
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
six.reraise(typ, value, traceback)
File "/opt/conda/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3097, in train
saving_listeners=saving_listeners)
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 350, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1176, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1209, in _train_model_default
saving_listeners)
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1515, in _train_with_estimator_spec
save_graph_def=self._config.checkpoint_save_graph_def) as mon_sess:
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 605, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1039, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 750, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1232, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1237, in _create_session
return self._sess_creator.create_session()
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 910, in create_session
hook.after_create_session(self.tf_sess, self.coord)
File "/opt/conda/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/util.py", line 94, in after_create_session
session.run(self._initializers)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 968, in run
run_metadata_ptr)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1191, in _run
feed_dict_tensor, options, run_metadata)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1369, in _do_run
run_metadata)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.InvalidArgumentError: From /job:worker/replica:0/task:0:
NodeDef mentions attr 'return_nbest' not in Op<name=SentencepieceTokenizeOp; signature=sp_handle:resource, input:string, nbest_size:int32, alpha:float, add_bos:bool, add_eos:bool, reverse:bool -> output_values:out_type, output_splits:Tsp
lits; attr=out_type:type,default=DT_INT32,allowed=[DT_INT32, DT_STRING]; attr=Tsplits:type,default=DT_INT64,allowed=[DT_INT32, DT_INT64]; is_stateful=true>; NodeDef: {{node SentenceTokenizer/SentenceTokenizer/SentencepieceTokenizeOp}}. (
Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[[SentenceTokenizer/SentenceTokenizer/SentencepieceTokenizeOp]]
[[MakeIterator]]
In call to configurable 'run' (<function run at 0x7faa6757c560>)
If it helps, here are all the package versions:
absl-py 0.12.0
aiohttp 3.7.4.post0
ansiwrap 0.8.4
anyio 3.3.0
appdirs 1.4.4
argcomplete 1.12.3
argon2-cffi 20.1.0
arrow 1.1.1
asn1crypto 1.4.0
astunparse 1.6.3
async-generator 1.10
async-timeout 3.0.1
attrs 21.2.0
avro-python3 1.9.2.1
Babel 2.9.1
backcall 0.2.0
backports.entry-points-selectable 1.1.0
backports.functools-lru-cache 1.6.4
beatrix-jupyterlab 0.9.1
binaryornot 0.4.4
black 21.7b0
bleach 4.0.0
blinker 1.4
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 4.2.2
caip-notebooks-serverextension 1.0.0
certifi 2021.5.30
cffi 1.14.6
chardet 4.0.0
charset-normalizer 2.0.4
clang 5.0
click 8.0.1
cloud-tpu-client 0.10
cloud-tpu-profiler 2.4.0
cloudpickle 1.6.0
colorama 0.4.4
conda 4.10.3
conda-package-handling 1.7.3
confuse 1.5.0
cookiecutter 1.7.3
crcmod 1.7
cryptography 3.4.7
cycler 0.10.0
dataclasses 0.8
debugpy 1.4.1
decorator 5.0.9
defusedxml 0.7.1
dill 0.3.4
distlib 0.3.2
dm-tree 0.1.6
docker 5.0.0
docker-pycreds 0.4.0
docopt 0.6.2
entrypoints 0.3
explainable-ai-sdk 1.3.0
explainers 0.1
fastavro 1.4.4
fasteners 0.16.3
filelock 3.0.12
flatbuffers 1.12
fsspec 2021.7.0
future 0.18.2
gast 0.4.0
gcsfs 2021.7.0
gin-config 0.4.0
gitdb 4.0.7
GitPython 3.1.18
google-api-core 1.31.1
google-api-python-client 2.17.0
google-apitools 0.5.31
google-auth 1.34.0
google-auth-httplib2 0.1.0
google-auth-oauthlib 0.4.5
google-cloud-aiplatform 1.1.1
google-cloud-appengine-logging 0.1.4
google-cloud-audit-log 0.1.0
google-cloud-bigquery 2.24.0
google-cloud-bigquery-storage 2.6.3
google-cloud-bigtable 2.3.3
google-cloud-core 1.7.2
google-cloud-dataproc 2.5.0
google-cloud-datastore 2.1.6
google-cloud-dlp 1.0.0
google-cloud-firestore 2.2.0
google-cloud-kms 2.5.0
google-cloud-language 2.2.2
google-cloud-logging 2.6.0
google-cloud-monitoring 2.4.2
google-cloud-profiler 3.0.5
google-cloud-pubsub 1.7.0
google-cloud-scheduler 2.3.2
google-cloud-spanner 3.8.0
google-cloud-speech 2.7.0
google-cloud-storage 1.42.0
google-cloud-tasks 2.5.1
google-cloud-translate 3.3.2
google-cloud-videointelligence 2.3.2
google-cloud-vision 2.4.2
google-crc32c 1.1.2
google-pasta 0.2.0
google-resumable-media 1.3.3
googleapis-common-protos 1.53.0
greenlet 1.1.1
grpc-google-iam-v1 0.12.3
grpcio 1.38.1
grpcio-gcp 0.2.2
gviz-api 1.9.0
h5py 3.1.0
hdfs 2.6.0
htmlmin 0.1.12
httplib2 0.19.1
huggingface-hub 0.0.16
idna 2.10
ImageHash 4.2.1
imageio 2.9.0
importlib-metadata 4.6.4
importlib-resources 5.2.2
ipykernel 6.2.0
ipython 7.26.0
ipython-genutils 0.2.0
ipython-sql 0.3.9
ipywidgets 7.6.3
jedi 0.18.0
Jinja2 2.11.3
jinja2-time 0.2.0
joblib 1.0.1
json5 0.9.5
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.12
jupyter-console 6.4.0
jupyter-core 4.7.1
jupyter-http-over-ws 0.0.8
jupyter-server 1.10.2
jupyter-server-mathjax 0.2.3
jupyterlab 1.2.21
jupyterlab-git 0.11.0
jupyterlab-pygments 0.1.2
jupyterlab-server 1.2.0
jupyterlab-widgets 1.0.0
keras 2.6.0
Keras-Preprocessing 1.1.2
keras-tuner 1.0.3
kiwisolver 1.3.1
kt-legacy 1.0.4
kubernetes 18.20.0
libcst 0.3.20
llvmlite 0.36.0
Markdown 3.3.4
MarkupSafe 1.1.1
matplotlib 3.4.3
matplotlib-inline 0.1.2
mesh-tensorflow 0.1.19
missingno 0.4.2
mistune 0.8.4
multidict 5.1.0
multimethod 1.4
mypy-extensions 0.4.3
nb-conda 2.2.1
nb-conda-kernels 2.3.1
nbclient 0.5.4
nbconvert 6.1.0
nbdime 3.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.5
nltk 3.6.2
notebook 6.4.3
notebook-executor 0.2
numba 0.53.1
numpy 1.19.5
oauth2client 4.1.3
oauthlib 3.1.1
olefile 0.46
opt-einsum 3.3.0
packaging 21.0
pandas 1.3.2
pandas-profiling 3.0.0
pandocfilters 1.4.2
papermill 2.3.3
parso 0.8.2
pathspec 0.9.0
patsy 0.5.1
pexpect 4.8.0
phik 0.11.2
pickleshare 0.7.5
Pillow 8.3.1
pip 21.2.4
platformdirs 2.2.0
portalocker 2.3.2
poyo 0.5.0
prettytable 2.1.0
prometheus-client 0.11.0
promise 2.3
prompt-toolkit 3.0.19
proto-plus 1.19.0
protobuf 3.16.0
psutil 5.8.0
ptyprocess 0.7.0
pyarrow 5.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.7
pycosat 0.6.3
pycparser 2.20
pydantic 1.8.2
pydot 1.4.2
Pygments 2.10.0
PyJWT 2.1.0
pymongo 3.12.0
pyOpenSSL 20.0.1
pyparsing 2.4.7
pyrsistent 0.17.3
PySocks 1.7.1
python-dateutil 2.8.2
python-slugify 5.0.2
pytz 2021.1
tensorflow-metadata 1.2.0
tensorflow-probability 0.13.0rc0
tensorflow-serving-api 2.5.1
tensorflow-text 2.6.0
termcolor 1.1.0
terminado 0.10.1
testpath 0.5.0
text-unidecode 1.3
textwrap3 0.9.2
tfds-nightly 4.4.0.dev202109080110
threadpoolctl 2.2.0
tifffile 2021.8.8
tokenizers 0.10.3
tomli 1.2.1
torch 1.9.0
tornado 6.1
tqdm 4.62.1
traitlets 5.0.5
transformers 4.10.0
typed-ast 1.4.3
typing-extensions 3.7.4.3
typing-inspect 0.7.1
ujson 4.0.2
Unidecode 1.2.0
uritemplate 3.0.1
urllib3 1.26.6
virtualenv 20.7.2
visions 0.7.1
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.57.0
Werkzeug 2.0.1
wheel 0.37.0
whichcraft 0.6.1
widgetsnbextension 3.5.1
witwidget 1.8.0
wrapt 1.12.1
xai-tabular-widget 0.1.0
yarl 1.6.3
zipp 3.5.0
Why does the error happen? Is this a package version issue?
craffel commented
Your tensorflow-text version is probably not matched to your tensorflow version (though I can't find tensorflow in your package list).
jiachangliu commented
@craffel I have uninstalled and reinstalled tensorflow and tensorflow-text, but the same error still occurs.
pip list gives me:
absl-py 0.12.0
aiohttp 3.7.4.post0
ansiwrap 0.8.4
anyio 3.3.0
apache-beam 2.32.0
appdirs 1.4.4
argcomplete 1.12.3
argon2-cffi 20.1.0
arrow 1.1.1
asn1crypto 1.4.0
astunparse 1.6.3
async-generator 1.10
async-timeout 3.0.1
attrs 21.2.0
avro-python3 1.9.2.1
Babel 2.9.1
backcall 0.2.0
backports.entry-points-selectable 1.1.0
backports.functools-lru-cache 1.6.4
beatrix-jupyterlab 0.9.1
binaryornot 0.4.4
black 21.7b0
bleach 4.0.0
blinker 1.4
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 4.2.2
caip-notebooks-serverextension 1.0.0
certifi 2021.5.30
cffi 1.14.6
chardet 4.0.0
charset-normalizer 2.0.4
clang 5.0
click 8.0.1
cloud-tpu-client 0.10
cloud-tpu-profiler 2.4.0
cloudpickle 1.6.0
colorama 0.4.4
conda 4.10.3
conda-package-handling 1.7.3
confuse 1.5.0
cookiecutter 1.7.3
crcmod 1.7
cryptography 3.4.7
cycler 0.10.0
dataclasses 0.8
debugpy 1.4.1
decorator 5.0.9
defusedxml 0.7.1
dill 0.3.1.1
distlib 0.3.2
dm-tree 0.1.6
docker 5.0.0
docker-pycreds 0.4.0
docopt 0.6.2
entrypoints 0.3
explainable-ai-sdk 1.3.0
explainers 0.1
fastavro 1.4.4
fasteners 0.16.3
filelock 3.0.12
flatbuffers 1.12
fsspec 2021.7.0
future 0.18.2
gast 0.4.0
gcsfs 2021.7.0
gin-config 0.4.0
gitdb 4.0.7
GitPython 3.1.18
google-api-core 1.31.1
google-api-python-client 1.8.0
google-apitools 0.5.31
google-auth 1.34.0
google-auth-httplib2 0.1.0
google-auth-oauthlib 0.4.5
google-cloud-aiplatform 1.1.1
google-cloud-appengine-logging 0.1.4
google-cloud-audit-log 0.1.0
google-cloud-bigquery 2.24.0
google-cloud-bigquery-storage 2.6.3
google-cloud-bigtable 1.7.0
google-cloud-core 1.7.2
google-cloud-dataproc 2.5.0
google-cloud-datastore 1.15.3
google-cloud-dlp 1.0.0
google-cloud-firestore 2.2.0
google-cloud-kms 2.5.0
google-cloud-language 1.3.0
google-cloud-logging 2.6.0
google-cloud-monitoring 2.4.2
google-cloud-profiler 3.0.5
google-cloud-pubsub 1.7.0
google-cloud-recommendations-ai 0.2.0
google-cloud-scheduler 2.3.2
google-cloud-spanner 1.19.1
google-cloud-speech 2.7.0
google-cloud-storage 1.42.0
google-cloud-tasks 2.5.1
google-cloud-translate 3.3.2
google-cloud-videointelligence 1.16.1
google-cloud-vision 1.0.0
google-crc32c 1.1.2
google-pasta 0.2.0
google-resumable-media 1.3.3
googleapis-common-protos 1.53.0
greenlet 1.1.1
grpc-google-iam-v1 0.12.3
grpcio 1.38.1
grpcio-gcp 0.2.2
gviz-api 1.9.0
h5py 3.1.0
hdfs 2.6.0
htmlmin 0.1.12
httplib2 0.19.1
huggingface-hub 0.0.16
idna 2.10
ImageHash 4.2.1
imageio 2.9.0
importlib-metadata 4.6.4
importlib-resources 5.2.2
ipykernel 6.2.0
ipython 7.26.0
ipython-genutils 0.2.0
ipython-sql 0.3.9
ipywidgets 7.6.3
jedi 0.18.0
Jinja2 2.11.3
jinja2-time 0.2.0
joblib 1.0.1
json5 0.9.5
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.12
jupyter-console 6.4.0
jupyter-core 4.7.1
jupyter-http-over-ws 0.0.8
jupyter-server 1.10.2
jupyter-server-mathjax 0.2.3
jupyterlab 1.2.21
jupyterlab-git 0.11.0
jupyterlab-pygments 0.1.2
jupyterlab-server 1.2.0
jupyterlab-widgets 1.0.0
keras 2.6.0
Keras-Preprocessing 1.1.2
keras-tuner 1.0.3
kiwisolver 1.3.1
kt-legacy 1.0.4
kubernetes 18.20.0
libcst 0.3.20
llvmlite 0.36.0
Markdown 3.3.4
MarkupSafe 1.1.1
matplotlib 3.4.3
matplotlib-inline 0.1.2
mesh-tensorflow 0.1.19
missingno 0.4.2
mistune 0.8.4
multidict 5.1.0
multimethod 1.4
mypy-extensions 0.4.3
nb-conda 2.2.1
nb-conda-kernels 2.3.1
nbclient 0.5.4
nbconvert 6.1.0
nbdime 3.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.5
nltk 3.6.2
notebook 6.4.3
notebook-executor 0.2
numba 0.53.1
numpy 1.19.5
oauth2client 4.1.3
oauthlib 3.1.1
olefile 0.46
opt-einsum 3.3.0
orjson 3.6.3
packaging 21.0
pandas 1.3.2
pandas-profiling 3.0.0
pandocfilters 1.4.2
papermill 2.3.3
parso 0.8.2
pathspec 0.9.0
patsy 0.5.1
pexpect 4.8.0
phik 0.11.2
pickleshare 0.7.5
Pillow 8.3.1
pip 21.2.4
platformdirs 2.2.0
portalocker 2.3.2
poyo 0.5.0
prettytable 2.1.0
prometheus-client 0.11.0
promise 2.3
prompt-toolkit 3.0.19
proto-plus 1.19.0
protobuf 3.16.0
psutil 5.8.0
ptyprocess 0.7.0
pyarrow 2.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.7
pycosat 0.6.3
pycparser 2.20
pydantic 1.8.2
pydot 1.4.2
Pygments 2.10.0
PyJWT 2.1.0
pymongo 3.12.0
pyOpenSSL 20.0.1
pyparsing 2.4.7
pyrsistent 0.17.3
PySocks 1.7.1
python-dateutil 2.8.2
python-slugify 5.0.2
pytz 2021.1
pyu2f 0.1.5
PyWavelets 1.1.1
PyYAML 5.4.1
pyzmq 22.2.1
qtconsole 5.1.1
QtPy 1.10.0
regex 2021.8.3
requests 2.25.1
requests-oauthlib 1.3.0
requests-unixsocket 0.2.0
retrying 1.3.3
rouge-score 0.0.4
rsa 4.7.2
ruamel-yaml-conda 0.15.100
sacrebleu 2.0.0
sacremoses 0.0.45
scikit-image 0.18.2
scikit-learn 0.24.2
scipy 1.7.1
seaborn 0.11.2
Send2Trash 1.8.0
sentencepiece 0.1.96
setuptools 57.4.0
simplejson 3.17.3
six 1.15.0
smmap 3.0.5
sniffio 1.2.0
SQLAlchemy 1.4.22
sqlparse 0.4.1
statsmodels 0.12.2
t5 0.6.2
tabulate 0.8.9
tangled-up-in-unicode 0.1.0
tenacity 8.0.1
tensorboard 2.6.0
tensorboard-data-server 0.6.1
tensorboard-plugin-profile 2.4.0
tensorboard-plugin-wit 1.8.0
tensorflow 2.6.0
tensorflow-cloud 0.1.14
tensorflow-datasets 4.3.0
tensorflow-estimator 2.6.0
tensorflow-hub 0.12.0
tensorflow-metadata 1.2.0
tensorflow-probability 0.13.0rc0
tensorflow-serving-api 2.6.0
tensorflow-text 2.6.0
tensorflow-transform 1.3.0
termcolor 1.1.0
terminado 0.10.1
testpath 0.5.0
text-unidecode 1.3
textwrap3 0.9.2
tfds-nightly 4.4.0.dev202109080110
tfx-bsl 1.3.0
threadpoolctl 2.2.0
tifffile 2021.8.8
tokenizers 0.10.3
tomli 1.2.1
torch 1.9.0
tornado 6.1
tqdm 4.62.1
traitlets 5.0.5
transformers 4.10.0
typed-ast 1.4.3
typing-extensions 3.7.4.3
typing-inspect 0.7.1
ujson 4.0.2
Unidecode 1.2.0
uritemplate 3.0.1
urllib3 1.26.6
virtualenv 20.7.2
visions 0.7.1
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.57.0
Werkzeug 2.0.1
wheel 0.37.0
whichcraft 0.6.1
widgetsnbextension 3.5.1
witwidget 1.8.0
wrapt 1.12.1
xai-tabular-widget 0.1.0
yarl 1.6.3
zipp 3.5.0
The error message is:
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 968, in run
run_metadata_ptr)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1191, in _run
feed_dict_tensor, options, run_metadata)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1369, in _do_run
run_metadata)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.InvalidArgumentError: From /job:worker/replica:0/task:0:
NodeDef mentions attr 'return_nbest' not in Op<name=SentencepieceTokenizeOp; signature=sp_handle:resource, input:string, nbest_size:int32, alpha:float, add_bos:bool, add_eos:bool, reverse:bool -> output_values:out_type, output_splits:Tsp
lits; attr=out_type:type,default=DT_INT32,allowed=[DT_INT32, DT_STRING]; attr=Tsplits:type,default=DT_INT64,allowed=[DT_INT32, DT_INT64]; is_stateful=true>; NodeDef: {{node SentenceTokenizer/SentenceTokenizer/SentencepieceTokenizeOp}}. (
Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[[SentenceTokenizer/SentenceTokenizer/SentencepieceTokenizeOp]]
[[MakeIterator]]
In call to configurable 'run' (<function run at 0x7f58660afe60>)
craffel commented
Is the TPU running 2.6.0 too?
jiachangliu commented
@craffel Could I ask how I can check whether TPU is running 2.6.0? I don't know how to do this.
Is tensorflow==2.6.0 not sufficient? I am using all flags recommended in the T5 GitHub page. Is there a flag I can use to specify TPU running on 2.6.0?
craffel commented
It's the --tf-version flag https://cloud.google.com/tpu/docs/ctpu-reference#up
jiachangliu commented
@craffel Now my flags after cput are as follows:
ctpu up --name=$TPU_NAME --project=$PROJECT --zone=$ZONE --tpu-size=$TPU_SIZE --tpu-only --noconf --tf-version=2.6.0
But the same error message still occurs.
To make sure I'm doing things correctly. I would like to check with you how I created the VM instance:
- For Series, I chose E2.
- For Machine type, I chose e2-standard-16.
- For Boot disk, I chose operating system "Deep Learning on Linux" and version "Deep Learning Image: TensorFlow Enterprise 2.6 m78 TPU/CPU".
Then for everything else including pip package installation, ctpu setup, and t5_mesh_transformer, I am running them inside the VM instance. This instance pops in a browser window after I clicked ssh-ing to the VM instance.
Is there anything else you could you think about that might cause this error message?
craffel commented
I don't know. I'd suggest asking on the TensorFlow support, since this doesn't seem to be an issue with the t5 library.