Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@jbms

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

Can this possibly go into a point release? I think it would be unfortunate for this bug to remain until 3.11, even though presumably it has been around since forever.

Can this possibly go into a point release? I think it would be unfortunate for this bug to remain until 3.11, even though presumably it has been around since forever.

The API deprecation must say 3.11. but the bugfix itself could be backported. We should let this one bake on the main branch for a bit first.

PS please do the CLA signing dance (see go/patching#cla internally, it's approved).

I already signed the CLA earlier today, may take until tomorrow to register.

I'm working on adding a mechanism that extension code can use to safely avoid threads hanging, and updating the documentation. I'll update this PR shortly once that is done.

There are legit use cases for pthread_exit(). If PyThread_exit_thread() is deprecated, does it mean that developers using it should call directly pthread_exit(0) and _endthreadex(0)?

I have updated the documentation to document the new functions, and changed to use SleepEx as suggested by @eryksun.

I have not yet made the change suggested by @gpshead to block signals in the pthread version --- I can do that if necessary, but I am not sure what the benefit is. Signal handlers already can't safely attempt to acquire the GIL or use any Python APIs that require the GIL, and therefore I would assume there is no harm in letting them run.

Other than that, what are the next steps for this pull request?

This PR is stale because it has been open for 30 days with no activity.

ping @eryksun @vstinner @gpshead

from your docs:

To avoid non-Python threads becoming blocked, or Python-created threads becoming
blocked while executing C extension code, you can use
:c:func:PyThread_TryAcquireFinalizeBlock and
:c:func:PyThread_ReleaseFinalizeBlock.

I think this phrasing undersells the magnitude of the problem and who can and should "fix" it... The "you can use" wording implies that it is the application user who debugged a hang is capable of resolving the problem by adjusting code they own.

But I suspect the reality is that until all Python C API using code in the world is updated to use these APIs (read: never/8-years)... hangs are going to be most frequently attributed to the huge pile of third party code that everybody's application uses. (Probably including a lot within the CPython internals itself).

There isn't really a solution for this. The existing GIL APIs and thus all existing Python C API using code in the world are not structured to check for and handle failure.

So it's mostly a matter of documentation semantics. We should seek to set expectations of who and when these new APIs should explicitly be used: In C code calling back into Python from potentially non-CPython created threads.

(I'd ignore the daemon thread aspect in our advice - anything could be one of those - we ultimately want to get rid of the daemon concept entirely and separately should document it as a "do not use / fundamentally broken / unfixable" feature in the threading module)

Other than the messaging, I think this overall PR looks good and want @Yhg1s to answer whether we backport to 3.12beta or not upon merge.

Not sure exactly what the best change to the docs is --- but as I see it, the key message around the hangs is:

"In most cases this is harmless"

I think it is a rare case indeed where the hang will not be harmless. Aside from the use case of wanting to cleanly finalize and initialize multiple times in a program, cases where hanging is a problem would likely result in a hang or crash without this PR as well.

I'm not sure I understand the point about PyGILState_Release() being confusing. I think the main question is whether there's a reasonable use-case for the finalize block outside of acquiring the GIL. For the use-case of making sure it's safe to acquire the GIL (and preventing finalization from starting while the GILState is held), I think the PyGILState API makes more sense. At that point it isn't exposed as a separate lock but as a new error condition (and a new safety guarantee when calling TryEnsure).

Framing it as "PyGILState_TryEnsure() is like PyGILState_Ensure() except it also prevents finalization from starting while the GILState is held" makes sense to me. I don't think most users of the C API care if their threads are blocked forever when finalizing, so they can just keep using PyGILState_Ensure. I don't think that's more confusing.

One example where it may be useful to acquire the finalize block separately from the GIL:

1. Acquire finalize block
2. Acquire a lock on some internal mutex owned by third-party code, finalize block ensures the thread doesn't hang while holding the mutex.
3. Do various things with the Python API, which may involve repeatedly acquiring and releasing the GIL
4. Release internal mutex
5. Release finalize block

In any case having a single enum and PyGILState_Release seems reasonable. I don't think TryEnsure is a great name to replace PyGILState_AcquireFinalizeBlockAndGIL because it doesn't clearly indicate that a finalize block is being acquired, and users should not acquire finalize blocks casually (e.g. while doing any sort of blocking operation), as that may unexpectedly prevent Python from exiting.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

I have made the requested changes; please review again

This PR can now be considered "on hold" or abandoned given the API questions regarding multiple interpreters. Please see the separate PR #105805 which contains just the minimal change to hang threads without adding any new public APIs.

Thanks for making the requested changes!

@gpshead, @ericsnowcurrently: please review the changes made to this pull request.

It looks like this PR was replaced by #105805, which has been merged now. Is there anything to be done here?