multiprocessing's default posix start method of `'fork'` is broken: change to ``'forkserver' || 'spawn'`
Closed this issue · 38 comments
| BPO | 40379 |
|---|---|
| Nosy | @pitrou, @mgorny, @Julian, @wimglenn, @applio, @itamarst |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2020-04-24.18:22:23.389>
labels = ['3.8', 'type-bug', '3.7', '3.9']
title = "multiprocessing's default start method of fork()-without-exec() is broken"
updated_at = <Date 2022-02-11.16:13:53.872>
user = 'https://bugs.python.org/itamarst'bugs.python.org fields:
activity = <Date 2022-02-11.16:13:53.872>
actor = 'mgorny'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2020-04-24.18:22:23.389>
creator = 'itamarst'
dependencies = []
files = []
hgrepos = []
issue_num = 40379
keywords = []
message_count = 11.0
messages = ['367210', '367211', '368173', '380478', '392358', '392501', '392503', '392506', '392507', '392508', '413081']
nosy_count = 8.0
nosy_names = ['pitrou', 'mgorny', 'Julian', 'wim.glenn', 'itamarst', 'davin', 'itamarst2', 'aduncan']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40379'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']Linked PRs
By default, multiprocessing uses fork() without exec() on POSIX. For a variety of reasons this can lead to inconsistent state in subprocesses: module-level globals are copied, which can mess up logging, threads don't survive fork(), etc..
The end results vary, but quite often are silent lockups.
In real world usage, this results in users getting mysterious hangs they do not have the knowledge to debug.
The fix for these people is to use "spawn" by default, which is the default on Windows.
Just a small sample:
- Today I talked to a scientist who spent two weeks stuck, until she found my article on the subject (https://codewithoutrules.com/2018/09/04/python-multiprocessing/). Basically multiprocessing locked up, doing nothing forever. Switching to "spawn" fixed it.
- dask/dask#3759 (comment) is someone who had issues fixed by "spawn".
- numpy/numpy#15973 is a NumPy issue which apparently impacted scikit-learn.
I suggest changing the default on POSIX to match Windows.
Looks like as of 3.8 this only impacts Linux/non-macOS-POSIX, so I'll amend the above to say this will also make it consistent with macOS.
Just got an email from someone for whom switching to "spawn" fixed a problem. Earlier this week someone tweeted about this fixing things. This keeps hitting people in the real world.
Another person with the same issue: https://twitter.com/volcan01010/status/1324764531139248128
I just ran into and fixed (thanks to itamarst's blog post) a problem likely related to this.
Multiprocessing workers performing work and sending a logging message back with success/fail info. I had a few intermittent deadlocks that became a recurring problem when I sped up the process that skipped tasks which had previously completed (I think this shortened the time between forking and attempting to send messages causing the third process to deadlock). After changing that it deadlocked *every time*.
Switching to "spawn" at the top of the main function has fixed it.
The problem with changing the default is that this will break any application that depends on passing non-picklable data to the child process (in addition to the potentially unexpected performance impact).
The docs already contain a significant elaboration on the matter, but feel free to submit a PR that would make the various caveats more explicit:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
This change was made on macOS at some point, so why not Linux? "spawn" is already the default on macOS and Windows.
The macOS change was required before "fork" simply ceased to work.
Windows has always used "spawn", because no other method can be implemented on Windows.
Given people's general experience, I would not say that "fork" works on Linux either. More like "99% of the time it works, 1% it randomly breaks in mysterious way".
Agreed, but again, changing will break some applications.
We could switch to forkserver, but we should have a transition period where a FutureWarning will be displayed if people didn't explicitly set a start method.
After updating PyPy3 to use Python 3.9's stdlib, we hit very bad hangs because of this — literally compiling a single file with "parallel" compileall could hang. In the end, we had to revert the change in how Python 3.9 starts workers because otherwise multiprocessing would be impossible to use:
https://foss.heptapod.net/pypy/pypy/-/commit/c594b6c48a48386e8ac1f3f52d4b82f9c3e34784
This is a very bad default and what's even worse is that it often causes deadlocks that are hard to reproduce or debug. Furthermore, since "fork" is the default, people are unintentionally relying on its support for passing non-pickleable projects and are creating non-portable code. The code often becomes complex and hard to change before they discover the problem.
Before we managed to figure out how to workaround the deadlocks in PyPy3, we were experimenting with switching the default to "spawn". Unfortunately, we've hit multiple projects that didn't work with this method, precisely because of pickling problems. Furthermore, they were surprised to learn that their code wouldn't work on macOS (in the end, many people perceive Python as a language for writing portable software).
Finally, back in 2018 I've made one of my projects do parallel work using multiprocessing. It gave its users great speedup but for some it caused deadlocks that I couldn't reproduce nor debug. In the end, I had to revert it. Now that I've learned about this problem, I'm wondering if this wasn't precisely because of "fork" method.
Another example: Nelson Elhage reports that "as of recently(?) pytorch silently deadlocks (even without GPUs involved at all) using method=fork so that's been fun to debug".
Examples he provided:
After updating a couple of libraries in a project we are working on, the code would hang without much explanation. After much debugging, I think one of the reasons for our issues is the forking default (this issue). Our business logic does not use multiprocessing, but the underlying execution engine does (in our case Luigi). Turns out that gRPC client (which was buried deep into one of our dependencies) can hang in some cases when forked grpc/grpc#18075. This was the case for us, and was very tricky to debug.
general plan:
- A DeprecationWarning in 3.12 and 3.13 when the default not-explicitly-specified start method of
forkis used on platforms where that is the default. - 3.14: flip the default for all platforms to
spawn.
Also worth noting that running without a GIL means fork() can start leaving Python data structures in a state that can lead to undefined behavior.
With a GIL, mutating a Python object can only happen in the Python thread holding the GIL, which will be the one calling fork(). In a non-GIL world, the locks might be per-object, which means thread 1 might be calling fork() at the same time as thread 2 is mutating a Python object of some sort.
Atomic-based lock-free data structures might also suffer from this problem? But trying to reason about that makes my head hurt 😬
Alright, the DeprecationWarning for default-fork is in. We're aiming to flip the default in 3.14.
I left it unspecified in the doc as to what the new default will be. I'd be happy with forkserver as it does perform better than spawn while addressing the primary threaded process compatibility issue. Lets make that decision during the 3.14-alpha phase.
I'm marking this closed as the process has been started. we can reopen in the 3.14 time frame to track completing the deprecation then.
hope you're ok with some early feedback -- I've been playing with the nightly and this change is very very annoying. even if code would work fine in a spawn-default world it is now riddled with un-actionable warnings (my code works fine whether or not it uses fork or spawn) -- I shouldn't need to set up a multiprocessing context just to silence some warnings when I've already done the due diligence to make my code work cross platform
Early feedback is exactly what we want, thanks! 😃 ❤️
Are the warnings being attributed to code you do not control? How are they un-actionable? Not wanting to take action is not the same as un-actionable.
If the warnings appear attributed to code you do not control and have no feedback channel into (bugs, PRs, etc), that could count. But these are DeprecationWarning, those are filtered by default. You should only be seeing them from unittests or __main__ code as those are contexts that imply "developer or code owner" where appropriate actions can be taken.
If you add an explicit multiprocessing.set_start_method call to your __main__ program the warning will go away:
import multiprocessing
multiprocessing.set_start_method("spawn")If you want to declare "forkserver" or "fork" for better performance when possible while keeping your code cross-platform safe, the logic probably looks like:
import multiprocessing, sys
multiprocessing.set_start_method("forkserver" if sys.platform not in ("darwin", "win32"))I don't like raw sys.platform values needing to exist in people's code. No-one should. But we have no way of explicitly saying "faster than spawn if possible please". If we added an ability to do that (not a bad idea), it would still be ugly for people supporting multiple Python versions:
import multiprocessing
try:
multiprocessing.set_start_method("faster-than-spawn-if-safe") # (A possible 3.14 default)
except ValueError: # Python versions before 3.12
multiprocessing.set_start_method("spawn") # always safeCaveat: I assume adding any of these would get tediously annoying to do within every single *_test.py file in a mp-heavy project. Which is why it's preferable to do it at the presumed-fewer multiprocessing/concurrent.futures call-sites themselves.
Put on a hat of someone who's code is not going to work after the start method changes. Instead of getting a warning to force them to acknowledge it and make their code's intent explicit, they'll suddenly be broken in a future release such as 3.14. We don't like treating our users that way when it can be avoided.
We effectively did that to people suddenly in 3.8 on macOS with it's change to 'spawn'. Because the platform broke so we had no choice in how to deal with the emergency there (see the long #77906 thread).
Some people do write code that depends on 'fork' sharing semantics for Linux+BSD, thus the deprecation period with a warning we're attempting to implement in this PR. We want people who depend on it to declare their dependency with an explicit "fork" specification. It'd be ideal to only warn "when needed", but there is no practical way to detect if somebody's code is relying on fork specific semantics.
The quiet alternative is to disable this new warning and have it be a documentation notice-only deprecation. That gives us an ability to smugly say "we told you so". But doesn't leave anyone who's code gets broken happy.
Which is less disruptive on the whole (not just to you)?
- A: Guaranteeing that some people requiring "fork" will have an unhappy surprise to debug in 3.14?
- B: Making more people annoyed in 3.12 that they're required to make their code's intent explicit.
I do realize that a consequence of this warning is that we're trying to force people into explicit is better than implicit use of the API during the transition period. It is hard to see being explicit about intent as a problem though.
Are the warnings being attributed to code you do not control? How are they un-actionable? Not wanting to take action is not the same as un-actionable.
it's both code I control and code I don't control. some of it is not wanting to take action, some of it I cannot control.
the root of it for me is, I have already done my due diligence to create cross platform software that works correctly given either default (mostly by nature of targetting windows (and now macos)) -- I should not be punished by a DeprecationWarning for doing so.
since I work on popular software, OS packagers with -Werror are going to be knocking on my door telling me my software is broken when in reality I've already done the hard work to ensure it is correct. I'll get drive-by (WRONG) PRs "following directions" from the warnings and forcing fork or spawn. this wastes my already limited time for open source on noise when I could be building cool new useful things
if the code today is correct and the code after 3.14 is correct, I shouldn't be getting a warning telling me to change it for 3.12 and 3.13
if I have to write a bunch of ugly code to re-introduce the default just so I don't get a bunch of annoying noisy issues / PRs for a DeprecationWarning that imo is wrong and that I don't care about I'm not happy
additionally all (except the ones specifically about a context type) of the documentation examples for multiprocessing will now fail with -Werror -- the documented way to do things should not be deprecated
I understand your frustration.
"If ... the code after 3.14 is correct" is impossible to detect and issue a warning about. That is what we want maintainers of code to manually verify and explicitly declare in their code.
How do you propose to get people to do this without a warning?
The root of the problem is that multiprocessing ever had a default in the first place - or at least that it wasn't the guaranteed safest method. (This mistake was made when pulling the original third party library in to become multiprocessing in ~2.6)
my assumption without data is that the vast majority of people have already made the necessary changes after 3.8 and that the warning is unnecessary
I suspect that's the best we can do. Apple's popularity means the majority of widely used things already dealt with start method compatibility? I'll remove the new warning.
my assumption without data is that the vast majority of people have already made the necessary changes after 3.8 and that the warning is unnecessary
I'm extremely skeptical about that. There's a lot of software that doesn't care about running on macOS or Windows.
I doubt it's more than a fraction of a percent of python users or pypi packages -- anything that people actually use has already been updated or is abandoned
If we remove the warning it follows that "some people requiring "fork" will have an unhappy surprise to debug in 3.14":
Here is one suggestion that might make their debugging experience slightly better: detect any pickling exception in ForkingPickler and print a helpful message that says "if your code used to work, then the change of default start method is likely the reason. Here is how to fix it..".
In my experience pickling has been the biggest source of incompatibilities between fork and spawn/forkserver.
Had this exact issue. I wrote custom script to train a Reinforcement Learning model using Tensorflow 2 on multiple process. The code works well on mac but unexpectedly hangs when I upload the code to my linux server. Waste multiple hours just to debug this. I'm happy the community is working to fix this issue!
This issue can cause duplicate uuids to be generated when using uuid1.
It looks like the first call to uuid1 sets up a global state, which is then copied to all processes when spawning a multiprocessing Pool.
Then all processes generate the same sequence of uuids.
See ClickHouse/clickhouse-connect#194 for more context on the issue and an example.
@guillaumematheron Which issue? The fact that fork is being used?
Regardless, uuid is an inefficient way to generate random ids.
Yes, as outlined in this comment, switching to spawn instead of fork prevents duplicate uuids from being generated.
Of course uuid1 is not a good way to generate random ids, since it's not random at all if a network address and counter are available.
But it should be safe to assume that it is unique, especially if it explicitly returns a flag saying that the value is "generated by the platform in a multiprocessing-safe way". Maybe it's worth adding a caveat to the uuid1 documentation ?
Please file a separate issue for the uuid module. It could be a reasonable decision to refresh global state like that upon fork, but that isn't going to happen buried in this issue. You could probably do it yourself today via os.register_at_fork().
The article is now https://pythonspeed.com/articles/python-multiprocessing/
This is in and done for 3.14 per our plan.
Thank you so much to everyone who worked on this!
Considering we recently have yet another case where changing fork to forkserver on Linux platforms causes hard-to-track bugs, I believe that we should:
- improve the changelog entry as suggested in #128145 (comment).
- mention this on the "highlighted" changes on https://www.python.org/downloads/release/ (at least for the rc and beta candidates)
- maybe more?
If docs improvements are needed, especially for users, I could work on that tomorrow. cc @gpshead
Documentation wise, what we need to do is improve the "What's New in 3.14" entry for multiprocessing. It doesn't currently go into details on user visible code behavior differences of the change. So people not already intimately familiar with the semantic consequences of the differences in multiprocessing start method behaviors ("there are dozens of us!") is currently left unaware.
Out of curiosity, did we get a pile of these bug reports with the release of Python 3.8? The start method was changed in 3.8 for macOS without even a deprecation period (because the platform gave us no choice). https://docs.python.org/3/whatsnew/3.8.html#multiprocessing
mention this on the "highlighted" changes on https://www.python.org/downloads/release/ (at least for the rc and beta candidates)
broken link, no idea what page you meant. We don't tend to list implementation details in a downloads page, many people do not get their Pythons (including alpha/beta/rcs) from such a place anyways. Downloads of versions link directly to What's New which is our canonical doc of important highlights for any given release.
feel free to work up a docs change and loop me in on the PR. :) I think the big one to highlight is a human level explanation of what will be accessible in the child process code. Ideally without requiring the reader to understand pickle either. 😅
broken link, no idea what page you meant.
Ah sorry, I meant the following for instance: https://www.python.org/downloads/release/python-3131/. We have highlights of what changed with this version and not everyone looks at the docs when downloading the version (those highlights are the RM's responsibility I think?). I'll work on a docs PR now and tag you when I'm done.
Out of curiosity, did we get a pile of these bug reports with the release of Python 3.8?
I don't know :( I wasn't involved with Python at that time!