[13.0] maximum recursion depth exceeded while calling a Python object
Closed this issue · 15 comments
In some calls it looks like New Relic in combination with Odoo is creating a maximum recursion depth.
An example from the log:
2020-05-07 07:04:21,912 28917 ERROR ? newrelic.core.data_collector: Unexpected exception when attempting to register the agent with the data collector. Please report this problem to New Relic support for further investigation.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/newrelic/core/data_collector.py", line 981, in create_session
'preconnect', license_key)
File "/usr/local/lib/python3.6/dist-packages/newrelic/core/data_collector.py", line 733, in send_request
max_payload_size_in_bytes)
File "/usr/local/lib/python3.6/dist-packages/newrelic/core/data_collector.py", line 460, in send_request
verify=cert_loc)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/sessions.py", line 504, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connectionpool.py", line 518, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connectionpool.py", line 322, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connectionpool.py", line 727, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/util/ssl_.py", line 237, in ssl_wrap_socket
ciphers=ciphers)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/util/ssl_.py", line 207, in create_urllib3_context
context.options |= options
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connectionpool.py", line 518, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connectionpool.py", line 322, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connectionpool.py", line 727, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/util/ssl_.py", line 237, in ssl_wrap_socket
ciphers=ciphers)
File "/usr/local/lib/python3.6/dist-packages/newrelic/packages/requests/packages/urllib3/util/ssl_.py", line 207, in create_urllib3_context
context.options |= options
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 303 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
I've reported it to New Relic once but they basically said it had to do with gevent monkey patching. A quote from one of the New Relic support guys:
Thank you for your patience while we look further into this issue. I was able to speak to our developers about the recursion error you are seeing, and it turns out this error is not actually an error from the Python agent but is instead a known issue with gevent. You can read more about this error at the following link:
gevent/gevent#1531
I would highly recommend reviewing this issue and making the suggested workaround in the Github example
But Odoo is already patched at https://github.com/odoo/odoo/blob/4b12643abc6b8ba34f14e468e8d7760bc2db385b/odoo/__init__.py#L31
I'm not sure what's up or why it happens but I decided to create an issue for a start. Any help or feedback is appreciated :)
I had the same issue, fixed with pip install --upgrade gevent & pip install --upgrade requests
Tested it again today but still getting the recursion errors once in a while. Also after upgrading to the latest gevent and requests packages.
@jaredkipe I saw recently you talked about upgrading it & using it more again in #10 (comment). Did you happen to see the same behaviour around yet? I'd like to tackle it because I believe it also means some of the content is not monitored/sent to New Relic 🤔
Seeing this as well on an 11.0 deployment that has the gevent monkey patch.
gevent==1.1.2
requests==2.20.0
@jaredkipe any chance you could give us an update/feedback? 🤞
Found a comment on the newrelic support forums indicating v7 of the newrelic agent should be used.
@ChrisCalculus thanks for referencing this! Did you test it and did it resolve the issue for you?
@Yenthe666 I'm currently testing this (and haven't seen the error yet). I needed to make some changes to the plugin (visible in this PR) in order to ensure compatibility with the newer NewRelic plugin version.
@Yenthe666 @ChrisCalculus Sorry, haven't had time to improve or look at this module, so I have no feedback to give. I'm always interested in bumping requirements like gevent for good reasons.
@Yenthe666 we had the same issue, the system was just working fine for 2 years, trying to get some data from external url and create some entries
using response = requests.get(url) and some code to loop on the response.content
yesterday we recognised that no data was created for the 2 previous dates and when we checked the cron result
we get the error "maximum recursion depth exceeded while calling a Python object"
What are we on these days, 15 - right?! I'm gonna get this working and work my way backwards in versions.
What are we on these days, 15 - right?! I'm gonna get this working and work my way backwards in versions.
Yes, V16 in 1.5 month 😉
Alright, pushed 15.0.0.9.0 here and have personally tested it with threaded and workers and with multiple nodes pushed up over 2k rpm throughput (k6 between / and /shop). I did straight-up-comment-out the whole controller class but it is good enough for me to use in principal and has been included in our current 15 docker image. You can get full pip package list via oneliner like docker run --rm -it hibou/hibou-odoo-suite:15.0 bash -c 'pip list' but for the particular packages discussed above (and newrelic):
gevent==20.9.0
requests==2.25.1
newrelic==8.0.0.179
I'd guess the 'upgrade gevent and requests' comment probably holds weight.
I don't know when I'll be able to start work on earlier versions, but I'd guess that the packages make more of a different, and maybe just comment out the controller ;)
I just tested my current build pipeline for 11 (hibou/hibou-odoo-suite:11.0), which does include this module, and it works perfectly fine for me (admittedly some lack of info compared to current packages). But I probably need a better way of reproducing it.... Can someone make a server action that reproduces the issue? Maybe odoo shell code (so you can import requests)?
gevent==1.5.0
requests==2.27.1
newrelic==4.18.0.118
So packaging seems to be some cause... Want to compare your pip list to the one in these docker images?
root@ee11-1--web-5d7454b6c6-l77mf:/# pip install --upgrade gevent==20.9.0
Collecting gevent==20.9.0
Downloading gevent-20.9.0-cp37-cp37m-manylinux2010_x86_64.whl (5.5 MB)
|████████████████████████████████| 5.5 MB 36.3 MB/s
Collecting zope.interface
Downloading zope.interface-5.4.0-cp37-cp37m-manylinux2010_x86_64.whl (251 kB)
|████████████████████████████████| 251 kB 56.1 MB/s
Collecting zope.event
Downloading zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from gevent==20.9.0) (53.0.0)
Collecting greenlet>=0.4.17
Downloading greenlet-1.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
|████████████████████████████████| 150 kB 59.9 MB/s
Installing collected packages: zope.interface, zope.event, greenlet, gevent
Attempting uninstall: greenlet
Found existing installation: greenlet 0.4.14
Uninstalling greenlet-0.4.14:
Successfully uninstalled greenlet-0.4.14
Attempting uninstall: gevent
Found existing installation: gevent 1.5.0
Uninstalling gevent-1.5.0:
Successfully uninstalled gevent-1.5.0
Successfully installed gevent-20.9.0 greenlet-1.1.3 zope.event-4.5.0 zope.interface-5.4.0
root@ee11-1--web-5d7454b6c6-l77mf:/# pip install --upgrade requests==2.25.1
Collecting requests==2.25.1
Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 14.3 MB/s
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests==2.25.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests==2.25.1) (2018.11.29)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests==2.25.1) (2.6)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests==2.25.1) (1.26.8)
Installing collected packages: requests
Attempting uninstall: requests
Found existing installation: requests 2.27.1
Uninstalling requests-2.27.1:
Successfully uninstalled requests-2.27.1
Successfully installed requests-2.25.1
Also works fine, which is a huge gevent upgrades and what I now realize is a downgrade on requests... (I should upgrade my requests in 15!)
I'm going to close this issue because it appears to have to do with packaging more than this module. OR I just don't understand where/how it is broken.