gh-109047: concurrent.futures catches RuntimeError#109810

vstinner

concurrent.futures: The 'executor manager thread' now catches
PythonFinalizationError, it calls terminate_broken(). The exception
occurs while Python is being finalized when adding an item to the
'call queue' tries to create a new 'queue feeder' thread.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

terminate_broken() no longer calls shutdown_workers() since the
queue is no longer working anymore (read and write ends of the
queue pipe are closed).
terminate_broken() now terminates child processes.
wait_result_broken_or_wakeup() now uses the short form of
traceback.format_exception().

multiprocessing.Queue changes:

Add _terminate_broken() method.
_start_thread() sets _thread to None on exception to prevent
leaking "dangling threads" even if the thread was not started
yet.

Issue: test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047

📚 Documentation preview 📚: https://cpython-previews--109810.org.readthedocs.build/

vstinner

This PR relies on PR #109809 which adds PythonFinalizationError exception.

test_concurrent_futures.test_python_finalization_error() is still unstable in my current draft implementation. Example:

ERROR: test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolForkserverProcessPoolExecutorTest.test_python_finalization_error)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_concurrent_futures/test_process_pool.py", line 209, in test_python_finalization_error
    list(executor.map(mul, [(2, 3)] * 10))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 850, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 821, in submit
    self._adjust_process_count()
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 780, in _adjust_process_count
    self._spawn_process()
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 798, in _spawn_process
    p.start()
  File "/home/vstinner/python/main/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/context.py", line 301, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
  File "/home/vstinner/python/main/Lib/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/connection.py", line 1173, in reduce_connection
    df = reduction.DupFd(conn.fileno())
                         ^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/connection.py", line 172, in fileno
    self._check_closed()
  File "/home/vstinner/python/main/Lib/multiprocessing/connection.py", line 138, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

vstinner

Another error, because a pipe file descriptor is closed:

ERROR: test_python_finalization_error (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_python_finalization_error)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_concurrent_futures/test_process_pool.py", line 209, in test_python_finalization_error
    list(executor.map(mul, [(2, 3)] * 10))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 850, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 821, in submit
    self._adjust_process_count()
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 780, in _adjust_process_count
    self._spawn_process()
  File "/home/vstinner/python/main/Lib/concurrent/futures/process.py", line 798, in _spawn_process
    p.start()
  File "/home/vstinner/python/main/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/vstinner/python/main/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/multiprocessing/util.py", line 453, in spawnv_passfds
    return _posixsubprocess.fork_exec(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: bad value(s) in fds_to_keep

…ueue() concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes. * wait_result_broken_or_wakeup() now uses the short form (1 argument, not 3) of traceback.format_exception(). * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet.

miss-islington

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

miss-islington

Sorry, @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 635184212179b0511768ea1cd57256e134ba2d75 3.12

…ython#109810) concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet. (cherry picked from commit 6351842)

bedevere-app

GH-110126 is a backport of this pull request to the 3.12 branch.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 macOS 3.x has failed when building commit 6351842.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/725/builds/5797) and take a look at the build logs.
Check if the failure is related to this commit (6351842) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/725/builds/5797

Failed tests:

test.test_concurrent_futures.test_deadlock

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 30, done.        
remote: Counting objects:   3% (1/30)        
remote: Counting objects:   6% (2/30)        
remote: Counting objects:  10% (3/30)        
remote: Counting objects:  13% (4/30)        
remote: Counting objects:  16% (5/30)        
remote: Counting objects:  20% (6/30)        
remote: Counting objects:  23% (7/30)        
remote: Counting objects:  26% (8/30)        
remote: Counting objects:  30% (9/30)        
remote: Counting objects:  33% (10/30)        
remote: Counting objects:  36% (11/30)        
remote: Counting objects:  40% (12/30)        
remote: Counting objects:  43% (13/30)        
remote: Counting objects:  46% (14/30)        
remote: Counting objects:  50% (15/30)        
remote: Counting objects:  53% (16/30)        
remote: Counting objects:  56% (17/30)        
remote: Counting objects:  60% (18/30)        
remote: Counting objects:  63% (19/30)        
remote: Counting objects:  66% (20/30)        
remote: Counting objects:  70% (21/30)        
remote: Counting objects:  73% (22/30)        
remote: Counting objects:  76% (23/30)        
remote: Counting objects:  80% (24/30)        
remote: Counting objects:  83% (25/30)        
remote: Counting objects:  86% (26/30)        
remote: Counting objects:  90% (27/30)        
remote: Counting objects:  93% (28/30)        
remote: Counting objects:  96% (29/30)        
remote: Counting objects: 100% (30/30)        
remote: Counting objects: 100% (30/30), done.        
remote: Compressing objects:   6% (1/16)        
remote: Compressing objects:  12% (2/16)        
remote: Compressing objects:  18% (3/16)        
remote: Compressing objects:  25% (4/16)        
remote: Compressing objects:  31% (5/16)        
remote: Compressing objects:  37% (6/16)        
remote: Compressing objects:  43% (7/16)        
remote: Compressing objects:  50% (8/16)        
remote: Compressing objects:  56% (9/16)        
remote: Compressing objects:  62% (10/16)        
remote: Compressing objects:  68% (11/16)        
remote: Compressing objects:  75% (12/16)        
remote: Compressing objects:  81% (13/16)        
remote: Compressing objects:  87% (14/16)        
remote: Compressing objects:  93% (15/16)        
remote: Compressing objects: 100% (16/16)        
remote: Compressing objects: 100% (16/16), done.        
remote: Total 16 (delta 13), reused 2 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '635184212179b0511768ea1cd57256e134ba2d75'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 6351842121 gh-109047: concurrent.futures catches PythonFinalizationError (#109810)
Switched to and reset branch 'main'

In file included from ./Modules/_tkinter.c:52:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
In file included from ./Modules/tkappinit.c:17:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
9 warnings generated.
9 warnings generated.

make: *** [buildbottest] Error 5

…110126) gh-109047: concurrent.futures catches PythonFinalizationError (#109810) concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet. (cherry picked from commit 6351842)

…ython#109810) concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise RuntimeError. Catch the exception and call terminate_broken() in this case. Add test_python_finalization_error() to test_concurrent_futures. concurrent.futures._ExecutorManagerThread changes: * terminate_broken() no longer calls shutdown_workers() since the call queue is no longer working anymore (read and write ends of the queue pipe are closed). * terminate_broken() now terminates child processes, not only wait until they complete. * _ExecutorManagerThread.terminate_broken() now holds shutdown_lock to prevent race conditons with ProcessPoolExecutor.submit(). multiprocessing.Queue changes: * Add _terminate_broken() method. * _start_thread() sets _thread to None on exception to prevent leaking "dangling threads" even if the thread was not started yet.

bedevere-app Bot mentioned this pull request Sep 25, 2023

test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047

Closed

vstinner force-pushed the cf_finalization_error branch from e286574 to 1722fb1 Compare September 25, 2023 00:25

vstinner mentioned this pull request Sep 25, 2023

gh-109370: Fix unexpected traceback output in test_concurrent_futures #109780

Merged

vstinner force-pushed the cf_finalization_error branch from 1722fb1 to 3137689 Compare September 29, 2023 18:20

vstinner marked this pull request as ready for review September 29, 2023 18:20

vstinner requested a review from rhettinger as a code owner September 29, 2023 18:20

bedevere-app Bot added the awaiting review label Sep 29, 2023

vstinner force-pushed the cf_finalization_error branch 2 times, most recently from a77c413 to a035765 Compare September 29, 2023 18:35

vstinner force-pushed the cf_finalization_error branch from a035765 to f94feb1 Compare September 29, 2023 18:37

vstinner mentioned this pull request Sep 29, 2023

gh-109047: Add PythonFinalizationError exception #109809

Closed

vstinner enabled auto-merge (squash) September 29, 2023 18:43

vstinner added the needs backport to 3.12 only security fixes label Sep 29, 2023

vstinner changed the title ~~gh-109047: concurrent.futures catches PythonFinalizationError~~ Sep 29, 2023

Fix Windows

2136c57

vstinner merged commit 6351842 into python:main Sep 29, 2023

vstinner deleted the cf_finalization_error branch September 29, 2023 19:31

bedevere-app Bot removed the awaiting review label Sep 29, 2023

miss-islington assigned vstinner Sep 29, 2023

bedevere-app Bot removed the needs backport to 3.12 label Sep 29, 2023

hugovk mentioned this pull request Sep 29, 2023

Dedent text to make code blocks copyable python/miss-islington#647

Merged

vstinner mentioned this pull request Nov 30, 2023

test_concurrent_futures.test_shutdown: AssertionError: b'' != b'apple' #112542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-109047: concurrent.futures catches RuntimeError#109810

gh-109047: concurrent.futures catches RuntimeError#109810
vstinner merged 2 commits into
python:mainfrom
vstinner:cf_finalization_error

vstinner commented Sep 25, 2023 •

edited

Loading

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

miss-islington commented Sep 29, 2023

Uh oh!

miss-islington commented Sep 29, 2023

Uh oh!

bedevere-app Bot commented Sep 29, 2023

Uh oh!

bedevere-bot commented Sep 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vstinner commented Sep 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

vstinner commented Sep 25, 2023

Uh oh!

miss-islington commented Sep 29, 2023

Uh oh!

miss-islington commented Sep 29, 2023

Uh oh!

bedevere-app Bot commented Sep 29, 2023

Uh oh!

bedevere-bot commented Sep 29, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vstinner commented Sep 25, 2023 •

edited

Loading