tfjs_next_page_prediction example failed
Bbean opened this issue · 1 comments
System information
- Have I specified the code to reproduce the issue: Yes
- Environment in which the code is executed : docker tfx latest(1.14.0.dev20230619)
- TensorFlow version:2.11.0
- TFX Version:1.12.0
- Python version:3.7
- Python dependencies (from
pip freeze
output):
absl-py==1.4.0
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1663850771047/work
aiohttp-cors==0.7.0
aiorwlock==1.3.0
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
ansiwrap==0.8.4
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1685486400275/work/dist
apache-beam==2.46.0
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work
argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1649500320262/work
astunparse==1.6.3
async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1640026696943/work
asynctest==0.13.0
attrs==21.4.0
Babel==2.12.1
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backoff==2.2.1
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
beatrix-jupyterlab @ file:///home/kbuilder/miniconda3/conda-bld/dlenv-tf-2-11-gpu_1686189831411/work/packages/beatrix_jupyterlab-2023.62.145605.tar.gz#sha256=dc0618129624f08938e5de9508880ef81dba8b272d097fd0b0c61f77af4da3f5
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1680888073205/work
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1674535352125/work
blessed==1.20.0
brotlipy==0.7.0
cached-property==1.5.2
cachetools==4.2.4
certifi==2023.5.7
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1666183775483/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1661170624537/work
chex==0.1.5
click==7.1.2
cloud-tpu-client==0.10
cloud-tpu-profiler==2.4.0
cloudpickle==2.2.1
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
colorful==0.5.5
conda==22.9.0
conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/work
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1669907009957/work
conda_package_streaming @ file:///home/conda/feedstock_root/build_artifacts/conda-package-streaming_1685101166527/work
crcmod==1.7
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1666563371538/work
cycler==0.11.0
Cython==0.29.35
dacite==1.8.1
db-dtypes==1.1.1
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1660619049122/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
Deprecated==1.2.14
dill==0.3.1.1
distlib==0.3.6
dm-tree==0.1.8
docker==4.4.4
docopt==0.6.2
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
etils==0.9.0
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1678703645500/work
explainable-ai-sdk==1.3.3
fastapi==0.96.0
fastavro==1.7.4
fasteners==0.18
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1684761244589/work/dist
filelock==3.12.0
flatbuffers==2.0.7
flax==0.6.4
flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1684084314667/work/source/flit_core
fonttools==4.38.0
frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1659556953511/work
fsspec==2023.1.0
gast==0.4.0
gcsfs==2023.1.0
gitdb==4.0.10
GitPython==3.1.31
google-api-core==1.32.0
google-api-python-client==1.8.0
google-apitools==0.5.31
google-auth==1.35.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-aiplatform==1.17.1
google-cloud-artifact-registry==1.8.1
google-cloud-bigquery==2.34.4
google-cloud-bigquery-storage==2.16.2
google-cloud-bigtable==1.7.3
google-cloud-core @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-core_1658759191372/work
google-cloud-datastore @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-datastore_1661336072891/work
google-cloud-dlp==3.9.2
google-cloud-language==1.3.2
google-cloud-monitoring==2.15.0
google-cloud-pubsub==2.13.11
google-cloud-pubsublite==1.6.0
google-cloud-recommendations-ai==0.7.1
google-cloud-resource-manager==1.6.3
google-cloud-spanner==3.26.0
google-cloud-storage==2.9.0
google-cloud-videointelligence==1.16.3
google-cloud-vision==3.1.4
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.5.0
googleapis-common-protos @ file:///home/conda/feedstock_root/build_artifacts/googleapis-common-protos-feedstock_1657690297642/work
gpustat==1.0.0
greenlet==2.0.2
grpc-google-iam-v1==0.12.6
grpcio @ file:///home/conda/feedstock_root/build_artifacts/grpc-split_1662408129152/work
grpcio-status @ file:///home/conda/feedstock_root/build_artifacts/grpcio-status_1662108958711/work
gviz-api==1.10.0
Gymnasium==0.26.3
gymnasium-notices==0.0.1
h11==0.14.0
h5py==3.8.0
hdfs==2.7.0
htmlmin==0.1.12
httplib2==0.21.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
ImageHash==4.3.1
imageio==2.31.0
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1653252814274/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1676919000169/work
iniconfig==2.0.0
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1666723258080/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1651240553635/work
ipython-genutils==0.2.0
ipython-sql==0.5.0
ipywidgets==7.7.5
jaraco.classes==3.2.3
jax==0.3.25
jaxlib==0.3.25
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
jeepney==0.8.0
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1654302431367/work
joblib==1.2.0
json5==0.9.14
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1669810440410/work
jupyter-http-over-ws==0.0.8
jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1671591499479/work
jupyter-server-mathjax==0.2.6
jupyter-server-proxy==3.2.2
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1673615989977/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1658332345782/work
jupyterlab==3.4.8
jupyterlab-git==0.41.0
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1649936611996/work
jupyterlab-widgets==1.1.4
jupyterlab_server==2.22.1
jupytext==1.14.6
keras==2.11.0
keras-tuner==1.3.5
keyring==23.13.1
keyrings.google-artifactregistry-auth==1.1.2
kiwisolver==1.4.4
kt-legacy==1.0.5
kubernetes==12.0.1
libclang==16.0.0
llvmlite==0.39.1
lz4==4.3.2
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.0.1
matplotlib==3.5.3
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mdit-py-plugins==0.3.5
mdurl==0.1.2
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1675771498296/work
ml-metadata==1.12.0
ml-pipelines-sdk==1.12.0
more-itertools==9.1.0
msgpack==1.0.5
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1648882415996/work
multimethod==1.9.1
nb-conda @ file:///home/conda/feedstock_root/build_artifacts/nb_conda_1654442778977/work
nb-conda-kernels @ file:///home/conda/feedstock_root/build_artifacts/nb_conda_kernels_1636999991206/work
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1683202085119/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1665125402713/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1683636231832/work
nbdime==3.2.0
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1679336765223/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
networkx==2.6.3
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1680870634737/work
notebook-executor @ file:///home/kbuilder/miniconda3/conda-bld/dlenv-tf-2-11-gpu_1686189831411/work/packages/notebook_executor
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1682360583588/work
numba==0.56.4
numpy==1.21.6
nvidia-ml-py==11.495.46
oauth2client==4.1.3
oauthlib==3.2.2
objsize==0.6.1
onnx==1.12.0
opencensus==0.11.2
opencensus-context==0.1.3
opentelemetry-api==1.18.0
opentelemetry-exporter-otlp==1.18.0
opentelemetry-exporter-otlp-proto-common==1.18.0
opentelemetry-exporter-otlp-proto-grpc==1.18.0
opentelemetry-exporter-otlp-proto-http==1.18.0
opentelemetry-proto==1.18.0
opentelemetry-sdk==1.18.0
opentelemetry-semantic-conventions==0.39b0
opt-einsum==3.3.0
optax==0.1.4
orbax==0.1.0
orjson==3.9.0
overrides==6.5.0
packaging==20.9
pandas==1.3.5
pandas-profiling==3.6.6
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
papermill==2.4.0
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
patsy==0.5.3
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
phik==0.12.3
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow==9.5.0
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1633981968097/work
platformdirs==3.5.1
plotly==5.14.1
pluggy==1.0.0
portpicker==1.5.2
prettytable==3.7.0
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1684970902412/work
promise==2.3
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1677600924538/work
proto-plus==1.22.2
protobuf==3.19.6
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1666155398032/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py-spy==0.3.14
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1666656960991/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydantic==1.10.9
pydot==1.4.2
pyfarmhash==0.3.2
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1681904169130/work
PyJWT==2.7.0
pymongo==3.13.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1685514481738/work
pyparsing==3.0.9
pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1649013358450/work
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
pytest==7.4.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
pytz==2023.3
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
PyWavelets==1.3.0
PyYAML==5.4.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1663830492333/work
ray==2.4.0
ray-cpp==2.4.0
regex==2023.6.3
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
requests-oauthlib==1.3.1
retrying==1.3.3
rich==13.4.1
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016701961/work
scikit-image==0.19.3
scikit-learn==1.0.2
scipy==1.7.3
seaborn==0.12.2
SecretStorage==3.3.3
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1682601222253/work
Shapely==1.8.5.post1
simpervisor==0.4
six @ file:///tmp/build/80754af9/six_1644875935023/work
smart-open==6.3.0
smmap==5.0.0
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1662051266223/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1658207591808/work
SQLAlchemy==2.0.15
sqlparse==0.4.4
starlette==0.27.0
statsmodels==0.13.5
tabulate==0.9.0
tangled-up-in-unicode==0.2.0
tenacity==8.2.2
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-profile==2.13.0
tensorboard-plugin-wit==1.8.1
tensorboardX==2.6
tensorflow==2.11.0
tensorflow-cloud==0.1.16
tensorflow-data-validation==1.12.0
tensorflow-datasets==4.8.2
tensorflow-decision-forests==1.2.0
tensorflow-estimator==2.11.0
tensorflow-hub==0.12.0
tensorflow-io==0.29.0
tensorflow-io-gcs-filesystem==0.29.0
tensorflow-metadata==1.12.0
tensorflow-model-analysis==0.43.0
tensorflow-probability==0.19.0
tensorflow-serving-api==2.11.0
tensorflow-transform==1.12.0
tensorflowjs==4.4.0
tensorstore==0.1.28
termcolor==2.3.0
terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1670253674810/work
textwrap3==0.9.2
tf2onnx==1.14.0
tfx==1.12.0
tfx-bsl==1.12.0
threadpoolctl==3.1.0
tifffile==2021.11.2
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
toml==0.10.2
tomli==2.0.1
toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1656937818679/work
tqdm @ file:///opt/conda/conda-bld/tqdm_1647339053476/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
typeguard==2.13.3
typer==0.9.0
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1685704949284/work
uritemplate==3.0.1
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1678635778344/work
uvicorn==0.22.0
virtualenv==20.21.0
visions==0.7.5
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1673864653149/work
webencodings==0.5.1
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1684708148666/work
Werkzeug==2.1.2
widgetsnbextension==3.6.4
witwidget==1.8.1
wordcloud==1.9.2
wrapt==1.15.0
wurlitzer==3.0.3
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1648966511831/work
ydata-profiling==4.2.0
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1677313463193/work
zstandard @ file:///home/conda/feedstock_root/build_artifacts/zstandard_1655887611100/work
Describe the current behavior
python tfjs_next_page_prediction_e2e_test.py
get error:
FAIL: testTFJSPagePredictionPipeline (main.TFJSNextPagePredictionPipelineEndToEndTest)
TFJSNextPagePredictionPipelineEndToEndTest.testTFJSPagePredictionPipeline
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 813, in _convert_tf_saved_model
resource_ids_maps = _get_resource_ids_maps(model, concrete_func, resource_initializer_concrete_func)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 680, in _get_resource_ids_maps
captured_input_index = resource_id_to_captured_input_index[_id]
KeyError: 3207
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/Documents/tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_e2e_test.py", line 94, in testTFJSPagePredictionPipeline
LocalDagRunner().run(pipeline)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/tfx_runner.py", line 124, in run
return self.run_with_ir(pipeline_pb, run_options=run_options_pb, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/local/local_dag_runner.py", line 109, in run_with_ir
component_launcher.launch()
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/launcher.py", line 573, in launch
executor_output = self._run_executor(execution_info)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/launcher.py", line 448, in _run_executor
executor_output = self._executor_operator.run_executor(execution_info)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/python_executor_operator.py", line 135, in run_executor
return run_with_executor(execution_info, executor)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/python_executor_operator.py", line 59, in run_with_executor
execution_info.exec_properties)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/executor.py", line 178, in Do
run_fn(fn_args)
File "/home/Documents/tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_util.py", line 198, in run_fn
rewriter.ModelType.TFJS_MODEL)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/converters.py", line 131, in rewrite_saved_model
dst_model_type)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/converters.py", line 47, in _invoke_rewriter
rewriter_inst.perform_rewrite(original_model, rewritten_model)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/rewriter.py", line 106, in perform_rewrite
self._rewrite(original_model, rewritten_model)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/tfjs_rewriter.py", line 91, in _rewrite
_ensure_str(rewritten_model.path))
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/tfjs_rewriter.py", line 29, in _convert_tfjs_model
saved_model_path, destination_path
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 818, in convert
weight_shard_size_bytes, metadata_map)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 539, in _dispatch_converter
metadata=metadata_map)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 988, in convert_tf_saved_model
frozen_graph_dir=frozen_graph_dir)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 820, in _convert_tf_saved_model
output_node_names)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 434, in _freeze_saved_model_v1
sess, meta_graph_def, output_node_names)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func
return func(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 322, in convert_variables_to_constants
variable_names_denylist=variable_names_blacklist)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1288, in convert_variables_to_constants_from_session_graph
variable_names_denylist=variable_names_denylist))
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py", line 943, in init
graph_def = graph_util.extract_sub_graph(graph_def, output_node_names)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func
return func(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 251, in extract_sub_graph
_assert_nodes_are_present(name_to_node, dest_nodes)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 204, in _assert_nodes_are_present
assert d in name_to_node, "%s is not in graph" % d
AssertionError: Identity is not in graph
Describe the expected behavior
tfjs testcase pass
Standalone code to reproduce the issue
tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_e2e_test.py
Any reply is welcome, thanks
This error looks like the pipeline fails at the tf model to tfjs model conversion step. I was able to replicate the issue. Both test case and pipeline fails with "AssertionError: Identity is not in graph" error.
Please find the attached gist. @briron, Could you please have a look why the tfjs_next_page_prediction example fails. Thanks