Issue 42639: Make atexit state per interpreter

Messages (7) msg382982 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-14 12:00

In Python 2.7, atexit was implemented in Python and registered itself using sys.exitfunc public attribute:
https://docs.python.org/2.7/library/sys.html#sys.exitfunc
https://docs.python.org/2.7/library/atexit.html#module-atexit

In Python 3.0, the atexit module was rewritten in C. A new private _Py_PyAtExit() function was added to set a new private global "pyexitfunc" variable: variable used by call_py_exitfuncs() called by Py_Finalize().

In Python 3.7, the global "pyexitfunc" variable was moved int _PyRuntimeState (commit 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6), and then into PyInterpreterState (commit 776407fe893fd42972c7e3f71423d9d86741d07c).

In Python 3.7, the atexit module was upgrade to the multiphase initialization API (PEP 489): PyInit_atexit() uses PyModuleDef_Init().

Since Python 2.7, the atexit module has a limitation: if a second instance is created, the new instance overrides the old one, and old registered callbacks are newer called.

One option is to disallow creating a second instance: see bpo-40600 and PR 23699 for that.

Another option is to move the atexit state (callbacks) into PyInterpreterState. Two atexit module instances would modify the same list of callbacks. In this issue, I propose to investigate this option.

msg383005 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-14 21:40

New changeset 83d52044ae4def1e8611a4b1b9263b850ca5c458 by Victor Stinner in branch 'master':
bpo-42639: Cleanup atexitmodule.c (GH-23770)
https://github.com/python/cpython/commit/83d52044ae4def1e8611a4b1b9263b850ca5c458

msg383008 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-14 22:08

New changeset 357704c9f2375f29ed5b3a93adac086fa714538d by Victor Stinner in branch 'master':
bpo-42639: atexit now logs callbacks exceptions (GH-23771)
https://github.com/python/cpython/commit/357704c9f2375f29ed5b3a93adac086fa714538d

msg383055 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-15 13:34

New changeset b8fa135908d294b350cdad04e2f512327a538dee by Victor Stinner in branch 'master':
bpo-42639: Move atexit state to PyInterpreterState (GH-23763)
https://github.com/python/cpython/commit/b8fa135908d294b350cdad04e2f512327a538dee

msg383064 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-15 15:08

New changeset f7049b5fb680c774e4950d10be62859a749f4e79 by Victor Stinner in branch 'master':
bpo-42639: Add script_helper.run_test_script() (GH-23777)
https://github.com/python/cpython/commit/f7049b5fb680c774e4950d10be62859a749f4e79

msg383071 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-15 16:12

New changeset 3ca2b8fd75043927f0bb03b8dac72d32beae255d by Victor Stinner in branch 'master':
bpo-42639: atexit._run_exitfuncs() uses sys.unraisablehook (GH-23779)
https://github.com/python/cpython/commit/3ca2b8fd75043927f0bb03b8dac72d32beae255d

msg383073 - (view) Author: STINNER Victor (vstinner) * Date: 2020-12-15 16:17

Ok, the initial issue is now fixed. It's possible to have more than one atexit module instance, registered callbacks in one or the other module are all called as expected. I also wrote an unit test for that.

I also took the opportunity of this issue to modernize the C code base, fix a bug in test_atexit (no longer clear atexit callbacks, run tests in subprocesses), and enhance atexit._run_exitfuncs(): log all callback exceptions using sys.unraisablehook so it's possible to catch them using the hook.

History Date User Action Args 2022-04-11 14:59:39adminsetgithub: 86805 2020-12-27 23:43:32vstinnerlinkissue40288 superseder 2020-12-15 16:17:11vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg383073

stage: patch review -> resolved

2020-12-15 16:12:10vstinnersetmessages: + msg383071 2020-12-15 15:11:37vstinnersetpull_requests: + pull_request22637 2020-12-15 15:08:24vstinnersetmessages: + msg383064 2020-12-15 14:11:31vstinnersetpull_requests: + pull_request22635 2020-12-15 14:09:30shihai1991setpull_requests: + pull_request22634 2020-12-15 14:09:22corona10setnosy: + corona10
pull_requests: + pull_request22633
2020-12-15 13:34:50vstinnersetmessages: + msg383055 2020-12-15 04:43:44shihai1991setnosy: + shihai1991
2020-12-14 22:08:19vstinnersetmessages: + msg383008 2020-12-14 21:43:33vstinnersetpull_requests: + pull_request22627 2020-12-14 21:40:47vstinnersetmessages: + msg383005 2020-12-14 21:18:56vstinnersetpull_requests: + pull_request22626 2020-12-14 12:02:08vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request22620 2020-12-14 12:00:33vstinnercreate