bpo-42972: Fully support GC for _winapi.Overlapped#26381
Conversation
|
Anyone knows a surefire way to test this? My main dev machine is Windows, but it seems that >>> import sys,gc
>>> for _ in range(5):
... print(sys.gettotalrefcount())
... import _winapi
... del sys.modules['_winapi']
... del _winapi
... gc.collect()
...
50468
0
51076
0
51432
0
51788
0
52144
0Not sure how to test just the |
Sorry, something went wrong.
|
Wait a minute, are multi phase init modules supposed to have a missing m_clear and m_traverse? https://github.com/python/cpython/blob/main/Modules/_winapi.c#L2046-L2052 |
Sorry, something went wrong.
Only if there's nothing to clear/traverse? :) |
Sorry, something went wrong.
there are over 100 objects created on module exec :(. Edit: also no m_free |
Sorry, something went wrong.
erlend-aasland
left a comment
There was a problem hiding this comment.
LGTM!
Sorry, something went wrong.
…for real this time ;-)
|
Ok seems things are better now after introducing the module cleanup functions: >>> import sys,gc
>>> for _ in range(5):
... print(sys.gettotalrefcount())
... import _winapi
... del sys.modules['_winapi']
... del _winapi
... gc.collect()
...
50524
42
50590
42
50590
42
50590
42
50590
42 |
Sorry, something went wrong.
|
That looks nice! If this is a reliable test method, we should use it, at least for the multi-init modules. |
Sorry, something went wrong.
Thanks. I shamelessly copied it from your message on the bpo. So thanks for the good test and the reviews :). |
Sorry, something went wrong.
I'm pretty sure I shamelessly stole it from @encukou: msg393508, issue 44116 😀 |
Sorry, something went wrong.
|
Regarding the tests. I propose to add a |
Sorry, something went wrong.
|
Thanks @Fidget-Spinner for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10. |
Sorry, something went wrong.
(cherry picked from commit 0fa282c) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
|
I'm not fully satisfied by the dealloc function which calls untrack late, but this code is already ugly and the evil |
Sorry, something went wrong.
Sorry i was going to move the untrack up, but timezones and work got in the way.
I hope so, Win7 is still pretty popular. WinXP is almost gone except for some niche areas if this site is to be believed. https://gs.statcounter.com/os-version-market-share/windows/desktop/worldwide Anyways, I appreciate the reviews and merge. Thanks! |
Sorry, something went wrong.
@erlend-aasland I think this is a worthwhile test to add in the future to make sure we don't regress (especially for module finalization reference leaks, I'm not too worried about the individual types with/without GC). Though I'm waiting for the dust to settle and see what the people on python-committers have to say about this issue specifically. (I'm also using it as an excuse to let everyone take a break. I think Victor has been working a little too hard reviewing all our PRs ;-). |
Sorry, something went wrong.
|
Thanks @Fidget-Spinner for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10. |
Sorry, something went wrong.
(cherry picked from commit 0fa282c) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
|
Thanks @Fidget-Spinner for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10. |
Sorry, something went wrong.
(cherry picked from commit 0fa282c) Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot AMD64 Fedora Stable 3.10 has failed when building commit 0d39951. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/all/#builders/659/builds/70 Failed tests:
Summary of the results of the build (if available): == Click to see traceback logsremote: Enumerating objects: 4, done.
remote: Counting objects: 25% (1/4)
remote: Counting objects: 50% (2/4)
remote: Counting objects: 75% (3/4)
remote: Counting objects: 100% (4/4)
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 50% (1/2)
remote: Compressing objects: 100% (2/2)
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 2), reused 2 (delta 2), pack-reused 0
From https://github.com/python/cpython
* branch 3.10 -> FETCH_HEAD
Note: switching to '0d399516320d8dfce4453037338659cef3a2adf4'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 0d39951632 [3.10] [bpo-42972](https://bugs.python.org/issue42972): Fully support GC for _winapi.Overlapped (GH-26381) (#26430)
Switched to and reset branch '3.10'
test_badargs (__main__.GeneralTest) ... ok
test_bound_methods (__main__.GeneralTest) ... ok
test_clear (__main__.GeneralTest) ... ok
test_exit (__main__.GeneralTest) ... ok
test_order (__main__.GeneralTest) ... ok
test_raise (__main__.GeneralTest) ... ok
test_raise_unnormalized (__main__.GeneralTest) ... ok
test_stress (__main__.GeneralTest) ... ok
test_unregister (__main__.GeneralTest) ... ok
----------------------------------------------------------------------
Ran 9 tests in 0.003s
OK
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... ok
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok
----------------------------------------------------------------------
Ran 24 tests in 8.272s
OK (skipped=2)
Fatal Python error: Segmentation fault
Current thread 0x00007fd746602640 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 325 in walk_tb
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 360 in extract
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 494 in __init__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 132 in format_exception
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 262 in handle_error
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2401 in wrap_conn
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2444 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1006 in _bootstrap_inner
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 963 in _bootstrap
Thread 0x00007fd74761f640 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1102 in _wait_for_tstate_lock
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1086 in join
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2604 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1006 in _bootstrap_inner
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 963 in _bootstrap
Thread 0x00007fd75a900740 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1102 in _wait_for_tstate_lock
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1086 in join
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2583 in __exit__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 3273 in test_ssl_cert_verify_error
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/case.py", line 549 in _callTestMethod
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/case.py", line 592 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/case.py", line 652 in __call__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/runner.py", line 176 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/support/__init__.py", line 959 in _run_suite
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/support/__init__.py", line 1082 in run_unittest
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 5007 in test_main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 141 in _runtest
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 194 in runtest
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest_mp.py", line 81 in run_tests_worker
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 661 in _main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 641 in main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 719 in main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/regrtest.py", line 43 in _main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/regrtest.py", line 47 in <module>
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/runpy.py", line 86 in _run_code
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/runpy.py", line 196 in _run_module_as_main
Extension modules: _testcapi (total: 1)
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_msilib skipped -- No module named '_msi'
test_winconsoleio skipped -- test only relevant on win32
test_winreg skipped -- No module named 'winreg'
test_devpoll skipped -- test works only on Solaris OS family
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_kqueue skipped -- test works only on BSD
test_winsound skipped -- No module named 'winsound'
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_startfile skipped -- object <module 'os' from '/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/os.py'> has no attribute 'startfile'
test_ioctl skipped -- Unable to open /dev/tty
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
Fatal Python error: Segmentation fault
Current thread 0x00007f9de2d1f640 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/linecache.py", line 72 in checkcache
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 375 in extract
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 494 in __init__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 132 in format_exception
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 262 in handle_error
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2401 in wrap_conn
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2444 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1006 in _bootstrap_inner
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 963 in _bootstrap
Thread 0x00007f9de0d1b640 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1102 in _wait_for_tstate_lock
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1086 in join
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2604 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1006 in _bootstrap_inner
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 963 in _bootstrap
Thread 0x00007f9df0b5c740 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1102 in _wait_for_tstate_lock
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/threading.py", line 1086 in join
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 2583 in __exit__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 3273 in test_ssl_cert_verify_error
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/case.py", line 549 in _callTestMethod
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/case.py", line 592 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/case.py", line 652 in __call__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 122 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/unittest/runner.py", line 176 in run
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/support/__init__.py", line 959 in _run_suite
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/support/__init__.py", line 1082 in run_unittest
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 5007 in test_main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 154 in _runtest
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/runtest.py", line 194 in runtest
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 321 in rerun_failed_tests
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 698 in _main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 641 in main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/libregrtest/main.py", line 719 in main
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/__main__.py", line 2 in <module>
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/runpy.py", line 86 in _run_code
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/runpy.py", line 196 in _run_module_as_main
Extension modules: _testcapi (total: 1)
make: *** [Makefile:1256: buildbottest] Segmentation fault (core dumped)
Cannot open file '/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/test-results.xml' for upload |
Sorry, something went wrong.
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot x86 Gentoo Installed with X 3.10 has failed when building commit 0d39951. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/all/#builders/696/builds/86 Failed tests:
Failed subtests:
Summary of the results of the build (if available): == Tests result: FAILURE then FAILURE == 413 tests OK. 1 test failed: 13 tests skipped: 1 re-run test: Total duration: 35 min 12 sec Click to see traceback logsTraceback (most recent call last):
File "/buildbot/buildarea/cpython/3.10.ware-gentoo-x86.installed/build/target/lib/python3.10/idlelib/idle_test/tkinter_testing_utils.py", line 54, in new_test_method
raise exception
File "/buildbot/buildarea/cpython/3.10.ware-gentoo-x86.installed/build/target/lib/python3.10/idlelib/idle_test/tkinter_testing_utils.py", line 38, in after_callback
next(test_generator)
File "/buildbot/buildarea/cpython/3.10.ware-gentoo-x86.installed/build/target/lib/python3.10/idlelib/idle_test/test_colorizer.py", line 586, in test_incremental_editing
eq(text.tag_nextrange('BUILTIN', '1.0'), ('1.0', '1.3'))
AssertionError: Tuples differ: () != ('1.0', '1.3')
Traceback (most recent call last):
File "/buildbot/buildarea/cpython/3.10.ware-gentoo-x86.installed/build/target/lib/python3.10/idlelib/idle_test/tkinter_testing_utils.py", line 54, in new_test_method
raise exception
File "/buildbot/buildarea/cpython/3.10.ware-gentoo-x86.installed/build/target/lib/python3.10/idlelib/idle_test/tkinter_testing_utils.py", line 38, in after_callback
next(test_generator)
File "/buildbot/buildarea/cpython/3.10.ware-gentoo-x86.installed/build/target/lib/python3.10/idlelib/idle_test/test_colorizer.py", line 569, in test_incremental_editing
eq(text.tag_nextrange('KEYWORD', '1.0'), ('1.0', '1.2'))
AssertionError: Tuples differ: () != ('1.0', '1.2')
|
Sorry, something went wrong.
https://bugs.python.org/issue42972