RecursionError when pickling Table object
Opened this issue · 0 comments
adinamarca commented
Hello.
Environment details
- OS type and version: Latest Mac OS and also in Linux, Ubuntu 24.04 amd/64
- Python version: 3.12.4
- pip version: 24.0
google-cloud-bigquery
version: 3.18.0
Steps to reproduce
- Create a valid instance of Table object.
- Using pickle, dumps the Table object.
- Using pickle, loads the Table object.
Code example
from warehouse.connectors import AccountAnalytics4Adminapi
from pickle import dumps, loads
def test_pickle():
conn = AccountAnalytics4Adminapi("someaccount")
conn.fetch()
conn.transform()
conn.load()
try:
pickled = dumps(conn)
unpickled = loads(pickled)
except Exception as e:
assert False, f"Failed to pickle/unpickle object: {e}"
assert conn == unpickled, "Pickled and unpickled objects are not equal."
Stack trace
Traceback (most recent call last):
File "/Users/some_user/Desktop/proyectos/warehouse/repo/warehouse/tests/test_pickle.py", line 12, in test_pickle
unpickled = loads(pickled)
^^^^^^^^^^^^^^
File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
value = self._xxx_field_to_index.get(name)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
value = self._xxx_field_to_index.get(name)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
value = self._xxx_field_to_index.get(name)
^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/some_user/Desktop/proyectos/warehouse/repo/tests.py", line 3, in <module>
test_pickle.test_pickle()
File "/Users/some_user/Desktop/proyectos/warehouse/repo/warehouse/tests/test_pickle.py", line 14, in test_pickle
assert False, f"Failed to pickle/unpickle object: {e}"
^^^^^
AssertionError: Failed to pickle/unpickle object: maximum recursion depth exceeded
I was using Airflow, and in a task a maximum recursion depth exceeded exception was raised. Also I checked this in my personal computer.
Im not expert, but I was checking the code and I see that in the "Row" object (which probably is used by Table
), there is a recursion when calling __getattr__
because it uses the get
method from _xxx_field_to_index
, and the get
method also uses the same get method from _xxx_field_to_index
, which leads to recursion (my guess).
Thanks!