Deprecation Warnings
adamjstewart opened this issue · 15 comments
Describe the bug
tensorboardX is using deprecated features of protobuf. This requires every project that indirectly depends on protobuf to add the following hack to their pyproject.toml: #654.
Minimal runnable code to reproduce the behavior
$ python -Werror -c 'import tensorboardX'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/__init__.py", line 5, in <module>
from .torchvis import TorchVis
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/torchvis.py", line 10, in <module>
from .writer import SummaryWriter
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/writer.py", line 16, in <module>
from .comet_utils import CometLogger
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/comet_utils.py", line 7, in <module>
from .summary import _clean_tag
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/summary.py", line 12, in <module>
from .proto.summary_pb2 import Summary
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/summary_pb2.py", line 16, in <module>
from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/tensor_pb2.py", line 16, in <module>
from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 18, in <module>
DESCRIPTOR = _descriptor.FileDescriptor(
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 1034, in __init__
_Deprecated('FileDescriptor')
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 97, in _Deprecated
warnings.warn(
DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.Expected behavior
I would not expect tensorboardX to use deprecated features.
Screenshots
N/A
Environment
$ pip list|grep -E "torch|proto|tensor"
efficientnet-pytorch 0.7.1
protobuf 3.20.1
pytorch-lightning 1.8.5
pytorch-sphinx-theme 0.0.24
segmentation-models-pytorch 0.3.1
tensorboardX 2.5.1
torch 1.13.1
torchmetrics 0.11.0
torchvision 0.14.1Python environment
Python 3.10.8, Spack Environment
Additional context
This issue was first reported in #653 and ignored in #654. The correct solution is to not use deprecated features.
Hi @adamjstewart, I was unable to reproduce the warning with your code. Here's my freshly created environment (with docker run -it --rm python:3.10 bash followed by some pip install):
Would you show your full environment?
Here's the full environment (collapsed so as not to take up the entire screen).
Click to expand
$ pip list
Package Version
----------------------------- -----------
affine 2.1.0
aiohttp 3.8.1
aiosignal 1.2.0
alabaster 0.7.12
antlr4-python3-runtime 4.8
anyio 3.6.1
appdirs 1.4.4
appnope 0.1.0
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.0.8
astunparse 1.6.3
async-timeout 4.0.2
attrs 22.1.0
Babel 2.10.3
backcall 0.2.0
beautifulsoup4 4.11.1
beniget 0.4.1
black 22.10.0
bleach 5.0.1
Bottleneck 1.3.5
build 0.7.0
cachetools 5.2.0
Cartopy 0.21.1
certifi 2022.9.14
cffi 1.15.1
cftime 1.0.3.4
charset-normalizer 2.0.12
click 8.1.3
click-plugins 1.1.1
cligj 0.5.0
cmocean 2.0
colorama 0.4.5
commonmark 0.9.1
contourpy 1.0.5
coverage 6.4.4
cycler 0.11.0
debugpy 1.6.3
decorator 5.1.1
defusedxml 0.7.1
docutils 0.19
efficientnet-pytorch 0.7.1
einops 0.6.0
entrypoints 0.4
et-xmlfile 1.0.1
executing 1.1.0
fastjsonschema 2.16.2
Fiona 1.8.22
flake8 5.0.4
fonttools 4.37.3
frozenlist 1.3.1
fsspec 2022.11.0
future 0.18.2
gast 0.5.3
GDAL 3.6.1
geocube 0.3.2
geopandas 0.11.1
gevent 1.5.0
greenlet 1.1.3
h5py 3.7.0
idna 3.4
imagesize 1.4.1
importlib-metadata 4.12.0
iniconfig 0.0.0
ipykernel 6.16.0
ipython 8.5.0
ipython-genutils 0.2.0
ipywidgets 8.0.2
isort 5.10.1
jaraco.classes 3.2.2
jdcal 1.3
jedi 0.18.1
Jinja2 3.1.2
joblib 1.2.0
json5 0.9.10
jsonschema 4.16.0
jupyter_client 7.3.5
jupyter_core 5.1.0
jupyter-server 1.21.0
jupyterlab 3.4.8
jupyterlab-pygments 0.2.2
jupyterlab-server 2.10.3
jupyterlab-widgets 3.0.3
keyring 23.9.1
kiwisolver 1.3.2
kornia 0.6.9
laspy 2.2.0
lightning-utilities 0.4.1
lxml 4.9.1
MarkupSafe 2.1.1
matplotlib 3.6.2
matplotlib-inline 0.1.6
mccabe 0.7.0
mercurial 5.8
mistune 2.0.4
more-itertools 8.14.0
multidict 6.0.2
munch 2.2.0
mypy 0.991
mypy-extensions 0.4.3
nbclassic 0.4.8
nbclient 0.6.7
nbconvert 7.0.0
nbformat 5.7.0
nbmake 1.3.3
nbsphinx 0.8.8
nest-asyncio 1.5.5
netCDF4 1.5.8
notebook 6.4.12
notebook_shim 0.2.2
numexpr 2.8.3
numpy 1.24.0
odc-geo 0.1.2
omegaconf 2.1.0
openpyxl 3.0.3
packaging 21.3
pandas 1.5.2
pandocfilters 1.5.0
parso 0.8.3
pathspec 0.10.1
pep517 0.12.0
pexpect 4.8.0
pickleshare 0.7.5
Pillow-SIMD 9.0.0.post1
pip 22.2.2
pkginfo 1.8.3
platformdirs 2.5.2
pluggy 1.0.0
ply 3.11
pretrainedmodels 0.7.4
prometheus-client 0.14.1
prompt-toolkit 3.0.31
protobuf 3.20.1
psutil 5.9.2
ptyprocess 0.7.0
pure-eval 0.2.2
py 1.11.0
pybind11 2.10.0
pycocotools 2.0.2
pycodestyle 2.9.1
pycparser 2.21
pydantic 1.10.2
pydocstyle 6.1.1
pyflakes 2.5.0
pygeos 0.10
Pygments 2.13.0
pyparsing 3.0.9
pyproj 3.3.1
pyrsistent 0.18.1
pyshp 2.1.0
pystac 1.4.0
pytest 7.1.3
pytest-cov 3.0.0
python-dateutil 2.8.2
pythran 0.11.0
pytorch-lightning 1.8.5
pytorch-sphinx-theme 0.0.24
pytz 2022.2.1
pyupgrade 2.31.1
PyYAML 6.0
pyzmq 24.0.1
radiant-mlhub 0.4.1
rarfile 4.0
rasterio 1.3.4
readme-renderer 37.3
requests 2.28.1
requests-toolbelt 0.9.1
rfc3986 1.4.0
rich 12.5.1
rioxarray 0.4.1.post0
Rtree 1.0.1
scikit-learn 1.2.0
scipy 1.9.3
SCons 4.3.0
segmentation-models-pytorch 0.3.1
Send2Trash 1.8.0
setuptools 59.4.0
Shapely 1.8.5
six 1.16.0
sniffio 1.3.0
snowballstemmer 2.2.0
snuggs 1.4.1
soupsieve 2.3.2.post1
Sphinx 5.3.0
sphinx-rtd-theme 0.5.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-programoutput 0.15
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data 0.5.0
tensorboardX 2.5.1
terminado 0.15.0
threadpoolctl 3.1.0
timm 0.4.12
tinycss2 1.1.1
tokenize-rt 4.2.1
toml 0.10.2
tomli 2.0.1
torch 1.13.1
torchmetrics 0.11.0
torchvision 0.14.1
tornado 6.2
tqdm 4.64.1
traitlets 5.7.1
twine 4.0.1
typing_extensions 4.3.0
urllib3 1.26.12
vermin 1.5.1
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.4.1
wheel 0.37.1
widgetsnbextension 4.0.3
xarray 2022.3.0
yarl 1.8.1
zipfile-deflate64 0.2.0
zipp 3.8.1If you revert #654 you should also be able to reproduce the issue in CI.
Thanks, I have found that I can reproduce with: PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python -Werror -c 'import tensorboardX'. This additional environment variable is also used in the test script so the deprecation message can be reproduced in CI.
Would you check if you have PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION set in your environment? By default, PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION is set to cpp, which should be faster.
I do not have that environment variable set, but both values error for me:
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python -Werror -c 'import tensorboardX'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/__init__.py", line 5, in <module>
from .torchvis import TorchVis
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/torchvis.py", line 10, in <module>
from .writer import SummaryWriter
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/writer.py", line 16, in <module>
from .comet_utils import CometLogger
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/comet_utils.py", line 7, in <module>
from .summary import _clean_tag
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/summary.py", line 12, in <module>
from .proto.summary_pb2 import Summary
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/summary_pb2.py", line 16, in <module>
from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/tensor_pb2.py", line 16, in <module>
from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 18, in <module>
DESCRIPTOR = _descriptor.FileDescriptor(
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 1034, in __init__
_Deprecated('FileDescriptor')
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 97, in _Deprecated
warnings.warn(
DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python -Werror -c 'import tensorboardX'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/__init__.py", line 5, in <module>
from .torchvis import TorchVis
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/torchvis.py", line 10, in <module>
from .writer import SummaryWriter
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/writer.py", line 16, in <module>
from .comet_utils import CometLogger
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/comet_utils.py", line 5, in <module>
from google.protobuf.json_format import MessageToJson
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/json_format.py", line 54, in <module>
from google.protobuf.internal import type_checkers
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/type_checkers.py", line 51, in <module>
from google.protobuf.internal import decoder
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/decoder.py", line 87, in <module>
from google.protobuf.internal import encoder
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/encoder.py", line 71, in <module>
from google.protobuf.internal import wire_format
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/wire_format.py", line 36, in <module>
from google.protobuf import descriptor
File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
from google.protobuf.pyext import _message
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/pyext/__init__.py)What about PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python -Werror -c "import google.protobuf"?
No errors there (tried google, google.protobuf, and google.protobuf.pyext).
Hi, I think the root cause is related to the protobuf in your spack env. Can you try PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python -Werror -c "from google.protobuf import __version__; print(__version__);from google.protobuf.pyext import _message" ?
According to File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module> from google.protobuf.pyext import _message This should trigger an error and print the actual protobuf used.
Yes, __version__ is 3.20.1, and the latter import shows the error.
Note that Spack installs separate packages for protobuf (cpp) and protobuf (python). My cpp implementation is version 3.21.7, while my python implementation is 3.20.1. Should those always match?
Thank you for telling me that your Spack env have different versions of protobuf installed. By convention, the major should be matched. However, as indicated in
https://developers.google.com/protocol-buffers/docs/news/2022-05-06
https://github.com/protocolbuffers/protobuf/releases/tag/v21.0-rc1
I think 3.21.7 (cpp) corresponds to the interface of 4.21.7 (python)
so the version mismatch of the python API feels like a major one. (3.20.1 -> 4.21.7)
Well there isn't a 4.X.Y (cpp) so I don't think it's possible to match major versions there. Do you mean minor versions?
Is it possible to uninstall both and just pip install protobuf==3.20.1?
I don't use pip. I can install different versions of cpp/python if necessary, just let me know which versions you want me to try.
I'm worried we're getting stuck down a rabbit hole here. If the issue also happens in CI, then it's not an issue with my installation, it's an issue with tensorboardX, right? Can we use the newer API instead?
Please make sure that protobuf 3.20.1 (python) is installed and it's using the 3.20.1 (cpp) backend.
In CI, the deprecation warning were shown because I forced the script to use the python backend. By default, the cpp backend should be used. As your experiment indicates, the error is more related to protobuf rather than tensorboardX. If you are interested in the new API, you can change the version and compile with
Line 6 in 8734a21