support for gunicorn workers
smiles3983 opened this issue · 16 comments
I would like to initialize the SDK once at the main application before my workers are spawned and have the SDK trace calls made by the workers. Currently, I have to initialize the SDK inside each worker, which gets complicated as to how many SDK instances I have running/shutdown.
We understand the problem and are in fact currently working on exposing the support for forked child processes that is present in the underlying C SDK to the Python SDK. So stay tuned!
Hi!
I'm also running gunicorn (in fact, many Django Deployments use gunicorn nowadays). Please let me know if I can be of any help, or if you need an environment to reproduce this.
Hi!
I'm also running gunicorn (in fact, many Django Deployments use gunicorn nowadays). Please let me know if I can be of any help, or if you need an environment to reproduce this.
Are you using the Python SDK for Dynatrace? Without it, when we run gunicorn directly, Dynatrace sees it, but we only get CPU and RAM usage, not anything deeper than that. My app uses bottle, and then calls gunicorn from within the framework. So I dont call gunicorn directly. But the main issue, is that when I have gunicorn use workers, I loose any dynatrace stuff. Even for the SDK, I have to set my tracing from within each worker. When I use the SDK from the master worker, I cant see anything going on inside the child workers. I end up with a ton of overhead having to initialize the SDK within each child worker. I know that Python 3.8 is supposed to start offering some sort of memory sharing ability between seperate processes. Im hoping between that and what Dynatrace can come up with might help out. At least with the SDK. The ideal solution would be to have Python get detected by dynatrace out of the box, but for now just have to work with the SDK.
Are you using the Python SDK for Dynatrace?
I'm using the python package oneagent-sdk
Without it, when we run gunicorn directly, Dynatrace sees it, but we only get CPU and RAM usage, not anything deeper than that.
yep, exactly. My understanding is that tracing requests through gunicorn is currently not working.
I am having the same issue trying to instrument celery workers
Just an update to let you know we are not forgetting this issue. We plan on releasing a new version of the Python SDK which exposes the fork support of the underlying SDK for C/C++ (https://github.com/Dynatrace/OneAgent-SDK-for-C#using-the-dynatrace-oneagent-sdk-with-forked-child-processes-only-available-on-linux).
We just released version 1.3.0 of the Python SDK, which contains fork support.
Please give it a try and let us now if that solved your issue.
By using this middleware https://github.com/Dynatrace/snippets/blob/master/technologies/python/flask/flaskr/dynatrace_middleware.py and by initializing the SDK (without forkable support) in every worker thread process (e.g., in Django you could do that in your wsgi.py, see example below) I was able to get it working.
wsgi.py
import oneagent
sdk_options = oneagent.sdkopts_from_commandline(remove=True)
init_result = oneagent.initialize(sdk_options)
application = DynatraceWSGIMiddleware(get_wsgi_application(), 'My App Name', 'my.host.com')Calling gunicorn with:
gunicorn --workers 4 --log-level debug --bind 127.0.1.1:5001 my_django_app.wsgiApparently this initializes the sdk 4 times (one time per worker thread of gunicorn), and this also worked with version 1.2 of the python oneagent sdk.
Now the last goal is to get this running with gunicorn and forkable, which should be doable using gunicorn configuration and the pre_fork function, but I still have to try that out.
https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/examples/example_config.py#L186-L190
Hi @christian-kreuzberger-dtx: When you talk about worker "threads", I think you mean worker processes. SDK initialization is always per-process, there is no way (or need) to initalize it per thread.
There is also no need to initialize the SDK in pre_fork: That would try to initialize the SDK for each worker process (though all but the first calls would be ignored by the SDK). It is enough to initialize the SDK once with forkable=true after the master process of the server started (just from skimming the docs, on_starting sounds like a good hook).
You are correct, I meant worker processes, sorry for the confusion.
trying to use the new forkable feature, and getting this error now:
File "/opt/app-root/lib/python3.6/site-packages/oneagent/__init__.py", line 253, in _try_init_noref
--
| sdk = nativeagent.initialize(sdklibname)
| File "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/nativeagent.py", line 35, in initialize
| return _force_initialize(loadsdk(sdkinit))
| File "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/sdkctypesiface.py", line 753, in loadsdk
| six.raise_from(SDKError(ErrorCode.LOAD_AGENT, msg), e)
| File "<string>", line 2, in raise_from
| oneagent.common.SDKError: (-1342308345, 'Failed loading SDK stub from /opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/libonesdk_shared.so: "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/libonesdk_shared.so: cannot open shared object file: No such file or directory"
Hi @smiles3983, this has most likely nothing to do with gunicorn or forkable, it looks like an installation issue. Try re-installing the oneagent-sdk package with -v --force and check if the output contains a big warning message with many ***. EDIT: This issue is actually explained in https://github.com/Dynatrace/OneAgent-SDK-for-Python#troubleshooting (CTRL+F libonesdk_shared.so).
Hi @smiles3983, this has most likely nothing to do with gunicorn or forkable, it looks like an installation issue. Try re-installing the oneagent-sdk package with
-v --forceand check if the output contains a big warning message with many***. EDIT: This issue is actually explained in https://github.com/Dynatrace/OneAgent-SDK-for-Python#troubleshooting (CTRL+F libonesdk_shared.so).
I added the reinstall after the pod starts up in openshift: same error:
Created wheel for oneagent-sdk: filename=oneagent_sdk-1.3.0.20191113.161138-py2.py3-none-manylinux1_x86_64.whl size=56973 sha256=d19848fd1c9fbcb8e4a7bf4cc9f5d66d9e17135192e4ba5d47c883b0ef652669
--
| Stored in directory: /tmp/pip-ephem-wheel-cache-r_jfvtfb/wheels/b0/6a/e0/0f43ee968490b1f0420f09540b0a0a16f902eca99b8e8f795f
| Successfully built oneagent-sdk
| Installing collected packages: oneagent-sdk
| Attempting uninstall: oneagent-sdk
| Found existing installation: oneagent-sdk 1.3.0.20191113.161138
| Uninstalling oneagent-sdk-1.3.0.20191113.161138:
| Successfully uninstalled oneagent-sdk-1.3.0.20191113.161138
| Successfully installed oneagent-sdk-1.3.0.20191113.161138
| "2020-01-24 13:50:48,478-insight-[__init__.py:227- initialize()]-DEBUG-initialize: ref count = 0"
| "2020-01-24 13:50:48,479-insight-[__init__.py:252- _try_init_noref()]-INFO-Initializing SDK with options=(), libname=None."
| Initializing SDK with options=(), libname=None.
| "2020-01-24 13:50:48,480-insight-[sdkctypesiface.py:746- loadsdk()]-INFO-Loading native SDK library "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/libonesdk_shared.so"."
| Loading native SDK library "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/libonesdk_shared.so".
| "2020-01-24 13:50:48,481-insight-[__init__.py:285- _try_init_noref()]-ERROR-Failed initializing agent."
| Traceback (most recent call last):
| File "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/sdkctypesiface.py", line 747, in loadsdk
| return SDKDllInterface(libname)
| File "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/sdkctypesiface.py", line 173, in __init__
| ERROR: Failed initializing agent.
| Traceback (most recent call last):
| File "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/sdkctypesiface.py", line 747, in loadsdk
| return SDKDllInterface(libname)
| File "/opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/sdkctypesiface.py", line 173, in __init__
| self._dll = ctypes.WinDLL(libname) if WIN32 else ctypes.CDLL(libname)
| File "/opt/rh/rh-python36/root/usr/lib64/python3.6/ctypes/__init__.py", line 343, in __init__
| self._handle = _dlopen(self._name, mode)
| OSError: /opt/app-root/lib/python3.6/site-packages/oneagent/_impl/native/libonesdk_shared.so: cannot open shared object file: No such file or directory
Are you sure you called pip with the -v (verbose) option?
FYI celery workers work fine with the new forkable keyword, thanks folks.
Closing this issue now, as it seems that the discussion is moving away from gunicorn-worker/fork specific issues.
@smiles3983 If you still need help, please open a new, more specific issue.