Python runtime deadlock due to logging + multithreading + fork

[varunbpatil]

Requests are being logged to sys.stdout using wsgi-request-logger.
The logging module (used by wsgi-request-logger) acquires a lock while writing to sys.stdout.
https://github.com/kubeless/runtimes/blob/b5fea6b3422166f13740089812a926497b6a5631/stable/python/kubeless.py#L105-L109

The default start method for a Process (using multiprocessing) in Unix is fork.

If a new Process is forked at exactly the same time as the sys.stdout lock being held in one of the threads of the main process, the lock state is replicated in the subprocess resulting in a deadlock when the subprocess tries to flush stdout (in _flush_std_streams) because the lock is already being held in the subprocess and there is nobody to release it.

  File "/opt/miniconda3/lib/python3.7/multiprocessing/util.py", line 400 in _flush_std_streams
  File "/opt/miniconda3/lib/python3.7/multiprocessing/process.py", line 317 in _bootstrap
  File "/opt/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 74 in _launch
  File "/opt/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20 in __init__
  File "/opt/miniconda3/lib/python3.7/multiprocessing/context.py", line 277 in _Popen
  File "/opt/miniconda3/lib/python3.7/multiprocessing/context.py", line 223 in _Popen
  File "/opt/miniconda3/lib/python3.7/multiprocessing/process.py", line 112 in start

This happens even without a single print statement in the function's handler.

More information about this issue can be found at the following links:

• https://bugs.python.org/issue28382
• https://bugs.python.org/issue6721

[andresmgot]

I am not a Python expert. What would be a possible solution here?

[varunbpatil]

Hi @andresmgot , I'm still trying to find a fix without losing any of the functionality we have today.

The recommended solution is to use the spawn or forkserver start method instead of fork.

The docs themselves say that fork is problematic when used with multithreading (cherrypy is a multithreaded server).

fork
The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.

But, the problem with spawn is that it is slower than fork and I also lose one very critical ability - the ability to load something into memory once and use that in all subsequent requests. For example, if I have a huge machine learning model to load from disk, today I can load it outside the handler function and it will be available to all the subsequent requests because forked processes start with a copy of the parent process' memory. If we do a spawn, each child process (i.e, request) will load the machine learning model from disk separately.

The problem with forkserver is that it may not be available on all platforms.

[varunbpatil]

Before #83 was fixed, we were never waiting for child processes to complete gracefully. The deadlock only occurs when we wait for the child process to complete (p.join()) and hence, this issue might have become more observable since the fix for #83 .

[varunbpatil]

Hi @andresmgot . Is there any plan to drop support for the Python2.7 runtime. Any solution using "spawn" or "forkserver" won't be possible in Python2.7 which only supports "fork".

[andresmgot]

Official support for python 2.7 has been dropped since Jan 2020 so I am okay if we need to remove it.