{{ message }}
gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it#107275
Merged
pitrou merged 5 commits intoAug 23, 2023
Merged
gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it#107275pitrou merged 5 commits into
pitrou merged 5 commits into
Conversation
Contributor
Sorry, something went wrong.
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
Sorry, something went wrong.
pitrou
reviewed
Aug 16, 2023
albanD
force-pushed
the
fix-issue-77377
branch
from
b6b25ff to
152800c
Compare
August 23, 2023 18:52
pitrou
reviewed
Aug 23, 2023
pitrou
added
needs backport to 3.11
only security fixes
needs backport to 3.12
only security fixes
labels
Aug 23, 2023
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 23, 2023
…ed Process before serializing it (pythonGH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 23, 2023
…ed Process before serializing it (pythonGH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
pitrou
added a commit
that referenced
this pull request
Aug 23, 2023
…sed Process before serializing it (GH-107275) (#108378) gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Yhg1s
pushed a commit
that referenced
this pull request
Aug 23, 2023
…sed Process before serializing it (GH-107275) (#108377) gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) Ensure multiprocessing SemLock is valid for spawn Process before serializing it. Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early. --------- (cherry picked from commit 1700d34) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Aug 30, 2023
… case (pythonGH-108568) pythongh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
pitrou
added a commit
that referenced
this pull request
Aug 30, 2023
…e case (GH-108568) (#108692) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Yhg1s
pushed a commit
that referenced
this pull request
Aug 30, 2023
…e case (GH-108568) (#108691) gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.
This fixes both the example in the initial report #77377 and other reports on the PyTorch repo.
Note that as far as I can tell, all objects that lead to segfault with mixed context are all caused by SemLock created with fork sent over a Spawn process. That's why it is the only object updated here and the only pair of ctx covered. Let me know if I'm mistaken and I can find a way to update other objects as well.
Also windows behavior is not changed here (and thus not tested).
Please let me know if the error message and/or the test are not appropriate and I can work on improving them.