BUG: AttributeError: 'int' object has no attribute 'split'
KrzysiekJa opened this issue · 6 comments
Describe the bug
Same error has occurred while using Neptune's Transformers integration during fine-tuning model with Transformers' Trainer.
Reproduction
I followed instructions available in your docs ( Transformers integration guide ). I used both proposed ways for enabling Neptune logging in my script that you're presenting on your page: using report_to="neptune" or instead of this just NeptuneCallback. Both resulted the same error.
Expected behavior
Traceback
https://app.neptune.ai/kjarek/sentence-similarity-research/e/SENSIM-219
/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/common/warnings.py:62: NeptuneUnsupportedType: You're attempting to log a type that is not directly supported by Neptune (<class 'NoneType'>).
Convert the value to a supported type, such as a string or float, or use stringify_unsupported(obj)
for dictionaries or collections that contain unsupported values.
For more, see https://docs.neptune.ai/help/value_of_unsupported_type
warnings.warn(
/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/common/warnings.py:62: NeptuneUnsupportedType: You're attempting to log a type that is not directly supported by Neptune (<class 'list'>).
Convert the value to a supported type, such as a string or float, or use stringify_unsupported(obj)
for dictionaries or collections that contain unsupported values.
For more, see https://docs.neptune.ai/help/value_of_unsupported_type
warnings.warn(
Traceback (most recent call last):
File "/net/tscratch/people/plgkjarek/Sentence-similarity-model-Barlow-Twins/sentence_similarity_neural_en.py", line 125, in <module>
trainer.train()
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/transformers/trainer.py", line 1862, in _inner_training_loop
self.control = self.callback_handler.on_train_begin(args, self.state, self.control)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/transformers/trainer_callback.py", line 353, in on_train_begin
return self.call_event("on_train_begin", args, state, control)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/transformers/trainer_callback.py", line 397, in call_event
result = getattr(callback, event)(
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/transformers/integrations.py", line 1348, in on_train_begin
self._log_model_parameters(model)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/transformers/integrations.py", line 1297, in _log_model_parameters
self._metadata_namespace[NeptuneCallback.model_parameters_key] = model.config.to_dict()
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/handler.py", line 117, in __setitem__
self[key].assign(value)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/handler.py", line 86, in inner_fun
return fun(self, *args, **kwargs)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/handler.py", line 211, in assign
self._container.define(self._path, value)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/metadata_containers/metadata_container.py", line 440, in define
attr.process_assignment(neptune_value, wait=wait)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/attributes/attribute.py", line 66, in process_assignment
return self.assign(value, wait=wait)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/attributes/namespace.py", line 109, in assign
self._container[f"{self._str_path}/{k}"].assign(v, wait=wait)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/handler.py", line 86, in inner_fun
return fun(self, *args, **kwargs)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/handler.py", line 211, in assign
self._container.define(self._path, value)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/metadata_containers/metadata_container.py", line 433, in define
neptune_value = cast_value(value)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/types/type_casting.py", line 86, in cast_value
return Namespace(value)
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/types/namespace.py", line 41, in __init__
empty_keys = [k for k in self.value.keys() if not parse_path(k)]
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/types/namespace.py", line 41, in <listcomp>
empty_keys = [k for k in self.value.keys() if not parse_path(k)]
File "/net/people/plgrid/plgkjarek/python_venv/lib64/python3.9/site-packages/neptune/internal/utils/paths.py", line 26, in parse_path
return _remove_empty_paths(path.split("/"))
AttributeError: 'int' object has no attribute 'split'
Shutting down background jobs, please wait a moment...
Done!
Environment
The output of pip list:
Package Version
------------------------ ----------
absl-py 1.4.0
accelerate 0.21.0
aiohttp 3.8.4
aiosignal 1.3.1
arrow 1.2.3
async-timeout 4.0.2
attrs 23.1.0
backoff 2.2.1
boto3 1.26.160
botocore 1.29.160
bravado 11.0.3
bravado-core 5.17.1
cachetools 5.3.1
certifi 2023.5.7
charset-normalizer 3.1.0
click 8.1.3
cmake 3.26.4
datasets 2.13.1
dill 0.3.6
docopt 0.6.2
filelock 3.12.2
fqdn 1.5.1
frozenlist 1.3.3
fsspec 2023.6.0
future 0.18.3
gitdb 4.0.10
GitPython 3.1.31
google-auth 2.21.0
google-auth-oauthlib 1.0.0
grpcio 1.56.0
huggingface-hub 0.15.1
idna 3.4
importlib-metadata 6.7.0
isoduration 20.11.0
Jinja2 3.1.2
jmespath 1.0.1
joblib 1.2.0
jsonpointer 2.4
jsonref 1.1.0
jsonschema 4.17.3
lit 16.0.6
Markdown 3.4.3
MarkupSafe 2.1.3
monotonic 1.6
mpmath 1.3.0
msgpack 1.0.5
multidict 6.0.4
multiprocess 0.70.14
neptune 1.3.2
networkx 3.1
nltk 3.8.1
numpy 1.25.0
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
nvidia-cufft-cu11 10.9.0.58
nvidia-curand-cu11 10.2.10.91
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusparse-cu11 11.7.4.91
nvidia-nccl-cu11 2.14.3
nvidia-nvtx-cu11 11.7.91
oauthlib 3.2.2
packaging 23.1
pandas 2.0.2
Pillow 9.5.0
pip 23.2
pipreqs 0.4.13
protobuf 3.20.3
psutil 5.9.5
pyarrow 12.0.1
pyasn1 0.5.0
pyasn1-modules 0.3.0
PyJWT 2.7.0
pyrsistent 0.19.3
python-dateutil 2.8.2
pytz 2023.3
PyYAML 6.0
regex 2023.6.3
requests 2.31.0
requests-oauthlib 1.3.1
rfc3339-validator 0.1.4
rfc3987 1.3.8
rsa 4.9
s3transfer 0.6.1
sacremoses 0.0.53
safetensors 0.3.1
scikit-learn 1.2.2
scipy 1.11.0
sentence-transformers 2.2.2
sentencepiece 0.1.99
setuptools 53.0.0
simplejson 3.19.1
six 1.16.0
smmap 5.0.0
swagger-spec-validator 3.0.3
sympy 1.12
tensorboard 2.13.0
tensorboard-data-server 0.7.1
threadpoolctl 3.1.0
tokenizers 0.13.3
torch 2.0.1
torchaudio 2.0.2
torchvision 0.15.2
tqdm 4.65.0
transformers 4.30.2
triton 2.0.0
typing_extensions 4.6.3
tzdata 2023.3
uri-template 1.3.0
urllib3 1.26.16
webcolors 1.13
websocket-client 1.6.1
Werkzeug 2.3.6
wheel 0.40.0
xxhash 3.2.0
yarg 0.1.9
yarl 1.9.2
zipp 3.15.0
The operating system you're using: Rocky Linux X.Y
The output of python --version: Python 3.9.14
Additional context
Add any other context about the problem here.
Maybe it will make things easier as I will also provide the leads for my script itself:
https://github.com/KrzysiekJa/Sentence-similarity-model-Barlow-Twins/blob/main/sentence_similarity_neural_en.py
Hello @KrzysiekJa 👋
Sorry that you are facing this issue 😔
Our engineering team is looking into this on priority.
Meanwhile, can you turn off logging parameters when you initialize NeptuneCallback()? This will remove the error, but trainer and model parameters will not be logged automatically.
In L92 of your script: neptune_callback = NeptuneCallback(log_parameters=False)
Please let me know if this is an acceptable workaround for you while we fix this 🙏
Hi @SiddhantSadangi :)
Your proposal has allowed my experiments begin to run successfully and be logged in Neptune. Thank you 🥇
Honestly, I'm not sure if the problem could be that I didn't notice the lack of pushing some changes related to the definition of compute_metrics function.
In L90 as predictions arg in metric.compute() I was using non existing cosine_scores and this could causing this NoneType problem. In that case sorry troubling 😔
Hey @KrzysiekJa , glad to be of help 🤗
We discovered that the issue is caused due to Neptune not accepting integers as namespace keys. model.config.to_dict() returns a dict under the key id2label that contains integer keys, which leads to the error. ❌
Nothing wrong with your usage here, at least nothing that could be causing the error you see.
This is on us, and we will get this fixed 🔧
I will keep this issue open till it is fixed.
Meanwhile, please feel free to contact us if there's anything you need assistance with. 🤗
@SiddhantSadangi , you're right, right now I experimented using NeptuneCallback without parameter log_parameters set to False and it didn't work ;)
Hey @KrzysiekJa ,
We just released neptune v1.6.0 which fixes this issue. 🚀
Once you update your version of Neptune, you should be able to log parameters as well ✅
Please let me know if this works for you!