gh-137400: Fix a crash when disabling profiling across all threads#137471

colesbury

The PyEval_SetProfileAllThreads function and other related functions had a race condition on tstate->c_profilefunc that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call.

There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads.

Issue: sys._setprofileallthreads race condition #137400

The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads.

colesbury

~~Closing in favor of the more complete PR: #137518~~

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @colesbury for commit 5e5e4f1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F137471%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

miss-islington-app

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

…ads (pythongh-137471) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads. (cherry picked from commit 3626928) Co-authored-by: Sam Gross <colesbury@gmail.com>

miss-islington-app

Sorry, @colesbury, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 362692852f13cdd1d33cc7ed35c0cbac7af1a785 3.13

bedevere-app

GH-137648 is a backport of this pull request to the 3.14 branch.

…ll threads (pythongh-137471) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads. (cherry picked from commit 3626928) Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-app

GH-137649 is a backport of this pull request to the 3.13 branch.

…eads (gh-137471) (gh-137649) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads. (cherry picked from commit 3626928)

…eads (gh-137471) (#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.14 (tier-3) has failed when building commit 4ebd928.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1794/builds/435) and take a look at the build logs.
Check if the failure is related to this commit (4ebd928) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1794/builds/435

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 24, done.        
remote: Counting objects:   4% (1/24)        
remote: Counting objects:   8% (2/24)        
remote: Counting objects:  12% (3/24)        
remote: Counting objects:  16% (4/24)        
remote: Counting objects:  20% (5/24)        
remote: Counting objects:  25% (6/24)        
remote: Counting objects:  29% (7/24)        
remote: Counting objects:  33% (8/24)        
remote: Counting objects:  37% (9/24)        
remote: Counting objects:  41% (10/24)        
remote: Counting objects:  45% (11/24)        
remote: Counting objects:  50% (12/24)        
remote: Counting objects:  54% (13/24)        
remote: Counting objects:  58% (14/24)        
remote: Counting objects:  62% (15/24)        
remote: Counting objects:  66% (16/24)        
remote: Counting objects:  70% (17/24)        
remote: Counting objects:  75% (18/24)        
remote: Counting objects:  79% (19/24)        
remote: Counting objects:  83% (20/24)        
remote: Counting objects:  87% (21/24)        
remote: Counting objects:  91% (22/24)        
remote: Counting objects:  95% (23/24)        
remote: Counting objects: 100% (24/24)        
remote: Counting objects: 100% (24/24), done.        
remote: Compressing objects:   7% (1/13)        
remote: Compressing objects:  15% (2/13)        
remote: Compressing objects:  23% (3/13)        
remote: Compressing objects:  30% (4/13)        
remote: Compressing objects:  38% (5/13)        
remote: Compressing objects:  46% (6/13)        
remote: Compressing objects:  53% (7/13)        
remote: Compressing objects:  61% (8/13)        
remote: Compressing objects:  69% (9/13)        
remote: Compressing objects:  76% (10/13)        
remote: Compressing objects:  84% (11/13)        
remote: Compressing objects:  92% (12/13)        
remote: Compressing objects: 100% (13/13)        
remote: Compressing objects: 100% (13/13), done.        
remote: Total 13 (delta 11), reused 0 (delta 0), pack-reused 0 (from 0)        
From https://github.com/python/cpython
 * branch                    3.14       -> FETCH_HEAD
Note: switching to '4ebd928b0096ca1dd78d5d3e6c787dbc40a43c11'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 4ebd928b009 [3.14] gh-137400: Fix a crash when disabling profiling across all threads (gh-137471) (#137648)
Switched to and reset branch '3.14'

configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  192k  100  192k    0     0  1474k      0 --:--:-- --:--:-- --:--:-- 1474k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 42455  100 42455    0     0   318k      0 --:--:-- --:--:-- --:--:--  318k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  6 5041k    6  333k    0     0  2019k      0  0:00:02 --:--:--  0:00:02 2019k
100 5041k  100 5041k    0     0  24.6M      0 --:--:-- --:--:-- --:--:--  135M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 1257k  100 1257k    0     0  9634k      0 --:--:-- --:--:-- --:--:-- 9634k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  635k  100  635k    0     0  5066k      0 --:--:-- --:--:-- --:--:-- 5066k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  479k  100  479k    0     0  2199k      0 --:--:-- --:--:-- --:--:-- 2199k
../../configure: line 4071: pkg-config: command not found
configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

../../Python/fileutils.c:460:1: warning: unused function 'decode_current_locale' [-Wunused-function]
  460 | decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen,
      | ^~~~~~~~~~~~~~~~~~~~~
../../Python/fileutils.c:679:1: warning: unused function 'encode_current_locale' [-Wunused-function]
  679 | encode_current_locale(const wchar_t *text, char **str,
      | ^~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
../../Modules/_localemodule.c:148:1: warning: unused function 'is_all_ascii' [-Wunused-function]
  148 | is_all_ascii(const char *str)
      | ^~~~~~~~~~~~
1 warning generated.
../../Modules/_hacl/Lib_Memzero0.c:66:6: warning: "Your platform does not support any safe implementation of memzero -- consider a pull request!" [-W#warnings]
   66 |     #warning "Your platform does not support any safe implementation of memzero -- consider a pull request!"
      |      ^
1 warning generated.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8784  100  8784    0     0  67630      0 --:--:-- --:--:-- --:--:-- 68093
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2894  100  2894    0     0   9811      0 --:--:-- --:--:-- --:--:--  9843
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 43504  100 43504    0     0   696k      0 --:--:-- --:--:-- --:--:--  708k

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.14 (tier-3) has failed when building commit 4ebd928.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1794/builds/436) and take a look at the build logs.
Check if the failure is related to this commit (4ebd928) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1794/builds/436

Summary of the results of the build (if available):

Click to see traceback logs

Note: switching to '4ebd928b0096ca1dd78d5d3e6c787dbc40a43c11'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 4ebd928b009 [3.14] gh-137400: Fix a crash when disabling profiling across all threads (gh-137471) (#137648)
Switched to and reset branch '3.14'

configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  192k  100  192k    0     0  1201k      0 --:--:-- --:--:-- --:--:-- 1201k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 42455  100 42455    0     0   419k      0 --:--:-- --:--:-- --:--:--  419k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 5041k  100 5041k    0     0  30.7M      0 --:--:-- --:--:-- --:--:-- 30.7M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 1257k  100 1257k    0     0  11.6M      0 --:--:-- --:--:-- --:--:-- 11.6M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  635k  100  635k    0     0  4794k      0 --:--:-- --:--:-- --:--:-- 4794k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  479k  100  479k    0     0  4090k      0 --:--:-- --:--:-- --:--:-- 4090k
../../configure: line 4071: pkg-config: command not found
configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

../../Python/fileutils.c:460:1: warning: unused function 'decode_current_locale' [-Wunused-function]
  460 | decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen,
      | ^~~~~~~~~~~~~~~~~~~~~
../../Python/fileutils.c:679:1: warning: unused function 'encode_current_locale' [-Wunused-function]
  679 | encode_current_locale(const wchar_t *text, char **str,
      | ^~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
../../Modules/_localemodule.c:148:1: warning: unused function 'is_all_ascii' [-Wunused-function]
  148 | is_all_ascii(const char *str)
      | ^~~~~~~~~~~~
1 warning generated.
../../Modules/_hacl/Lib_Memzero0.c:66:6: warning: "Your platform does not support any safe implementation of memzero -- consider a pull request!" [-W#warnings]
   66 |     #warning "Your platform does not support any safe implementation of memzero -- consider a pull request!"
      |      ^
1 warning generated.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8784  100  8784    0     0   142k      0 --:--:-- --:--:-- --:--:--  142k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2894  100  2894    0     0  55255      0 --:--:-- --:--:-- --:--:-- 55653
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 43504  100 43504    0     0   926k      0 --:--:-- --:--:-- --:--:--  944k

* main: pythongh-137288: Update 3.14 magic numbers (pythonGH-137665) pythongh-135228: When @DataClass(slots=True) replaces a dataclass, make the original class collectible (take 2) (pythonGH-137047) pythongh-126008: Improve docstrings for Tkinter cget and configure methods (pythonGH-133303) pythongh-131885: Use positional-only markers for ``max()`` and ``min()`` (python#131868) pythonGH-137426: Remove code deprecation of `importlib.abc.ResourceLoader` (pythonGH-137567) pythongh-125897: Mark range function parameters as positional only (python#125945) pythongh-137400: Fix a crash when disabling profiling across all threads (pythongh-137471) pythongh-115766: Fix IPv4Interface.is_unspecified (pythonGH-137326) pythongh-128813: cleanup C-API docs for PyComplexObject (pythonGH-137579) pythongh-135953: Profile a module or script with sampling profiler (python#136777) Fix documentation of hash in PyHash_FuncDef (python#137595)

hugovk

(These were temporary failures, fixed by a buildbot restart: #137186 (comment))

…ads (pythongh-137471) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads.

…ll threads (pythongh-137471) (python#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-app Bot mentioned this pull request Aug 6, 2025

sys._setprofileallthreads race condition #137400

Closed

godlygeek mentioned this pull request Aug 6, 2025

gh-137400: Stop the world when swapping profile functions #137472

Closed

colesbury closed this Aug 7, 2025

colesbury reopened this Aug 7, 2025

colesbury marked this pull request as ready for review August 7, 2025 18:55

bedevere-app Bot added the awaiting core review label Aug 7, 2025

colesbury added topic-free-threading needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Aug 7, 2025

colesbury requested review from hugovk and pablogsal August 7, 2025 18:57

colesbury added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Aug 7, 2025

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Aug 7, 2025

kumaraditya303 approved these changes Aug 11, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Aug 11, 2025

colesbury deleted the gh-137400-profileall branch August 11, 2025 15:41

bedevere-app Bot removed the awaiting merge label Aug 11, 2025

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Aug 11, 2025

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Aug 11, 2025

hugovk pushed a commit that referenced this pull request Aug 12, 2025

[3.14] gh-137400: Fix a crash when disabling profiling across all thr… …

4ebd928

…eads (gh-137471) (#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

kumaraditya303 pushed a commit to miss-islington/cpython that referenced this pull request Sep 9, 2025

[3.14] pythongh-137400: Fix a crash when disabling profiling across a… …

90dc6d8

…ll threads (pythongh-137471) (python#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-137400: Fix a crash when disabling profiling across all threads#137471

gh-137400: Fix a crash when disabling profiling across all threads#137471
colesbury merged 1 commit into
python:mainfrom
colesbury:gh-137400-profileall

colesbury commented Aug 6, 2025 •

edited by bedevere-app Bot

Loading

Uh oh!

colesbury commented Aug 7, 2025 •

edited

Loading

Uh oh!

bedevere-bot commented Aug 7, 2025

Uh oh!

Uh oh!

miss-islington-app Bot commented Aug 11, 2025

Uh oh!

miss-islington-app Bot commented Aug 11, 2025

Uh oh!

bedevere-app Bot commented Aug 11, 2025

Uh oh!

bedevere-app Bot commented Aug 11, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

Uh oh!

hugovk commented Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

colesbury commented Aug 6, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colesbury commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Aug 7, 2025

Uh oh!

Uh oh!

miss-islington-app Bot commented Aug 11, 2025

Uh oh!

miss-islington-app Bot commented Aug 11, 2025

Uh oh!

bedevere-app Bot commented Aug 11, 2025

Uh oh!

bedevere-app Bot commented Aug 11, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Aug 12, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

hugovk commented Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

colesbury commented Aug 6, 2025 •

edited by bedevere-app Bot

Loading

colesbury commented Aug 7, 2025 •

edited

Loading