GH-103906: Remove immortal refcounting in the interpreter#103909

brandtbucher

Issue: Avoid reference counting operations on None, True, False in the interpreter #103906

hauntsaninja

Hooray! Looks great

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @brandtbucher for commit fd95518 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

sobolevn

What about other places, like Modules/? Should it be done later?

JelleZijlstra

@sobolevn possibly, but it's best left to a different PR. In many places it's unlikely to make a significant difference, so it might not be worth the churn to make the change everywhere.

brandtbucher

Annoyingly, perf impact is neutral.

However, it appears that the results are being pulled way down by a couple of known unstable benchmarks that should be unaffected by this change. I might try running them again tonight, just to see.

(I still think this is worth doing, though, as it clearly removes unnecessary work from many hot code paths.)

markshannon

Even if the performance change is negligible, this seems like a worthwhile cleanup.

JelleZijlstra

I noticed one more opportunity like this: in the LIST_EXTEND opcode, none_val should always be None. Up to you whether that's worth including in this PR.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x has failed when building commit b4a9747.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/506/builds/4784) and take a look at the build logs.
Check if the failure is related to this commit (b4a9747) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/506/builds/4784

Failed tests:

test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 22, done.        
remote: Counting objects:   4% (1/22)        
remote: Counting objects:   9% (2/22)        
remote: Counting objects:  13% (3/22)        
remote: Counting objects:  18% (4/22)        
remote: Counting objects:  22% (5/22)        
remote: Counting objects:  27% (6/22)        
remote: Counting objects:  31% (7/22)        
remote: Counting objects:  36% (8/22)        
remote: Counting objects:  40% (9/22)        
remote: Counting objects:  45% (10/22)        
remote: Counting objects:  50% (11/22)        
remote: Counting objects:  54% (12/22)        
remote: Counting objects:  59% (13/22)        
remote: Counting objects:  63% (14/22)        
remote: Counting objects:  68% (15/22)        
remote: Counting objects:  72% (16/22)        
remote: Counting objects:  77% (17/22)        
remote: Counting objects:  81% (18/22)        
remote: Counting objects:  86% (19/22)        
remote: Counting objects:  90% (20/22)        
remote: Counting objects:  95% (21/22)        
remote: Counting objects: 100% (22/22)        
remote: Counting objects: 100% (22/22), done.        
remote: Compressing objects:  11% (1/9)        
remote: Compressing objects:  22% (2/9)        
remote: Compressing objects:  33% (3/9)        
remote: Compressing objects:  44% (4/9)        
remote: Compressing objects:  55% (5/9)        
remote: Compressing objects:  66% (6/9)        
remote: Compressing objects:  77% (7/9)        
remote: Compressing objects:  88% (8/9)        
remote: Compressing objects: 100% (9/9)        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 12 (delta 10), reused 4 (delta 3), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'b4a974792355cb0f525211b2962679e68949f02e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at b4a9747923 GH-103906: Remove immortal refcounting in the interpreter (GH-103909)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: drop_gil: PyMUTEX_LOCK(gil->switch_mutex) failed
Python runtime state: initialized

Thread 0x00007f03237a3b80 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 80 in clean_up_interpreters
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 88 in tearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 594 in _callTearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 637 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 690 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 240 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 88 in _run_code
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 198 in _run_module_as_main
Fatal Python error: Aborted

Thread 0x00007f03237a3b80 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 80 in clean_up_interpreters
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 88 in tearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 594 in _callTearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 637 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 690 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 240 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 88 in _run_code
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters (total: 2)
Aborted (core dumped)
make: *** [Makefile:1976: buildbottest] Error 134

Cannot open file '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/test-results.xml' for upload

* main: (26 commits) pythonGH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (python#104508) typing: Add more tests for TypeVar (python#104571) pythongh-104572: Improve error messages for invalid constructs in PEP 695 contexts (python#104573) typing: Use PEP 695 syntax in typing.py (python#104553) pythongh-102153: Start stripping C0 control and space chars in `urlsplit` (python#102508) pythongh-104469: Update README.txt for _testcapi (pythongh-104529) pythonGH-103092: isolate `_elementtree` (python#104561) pythongh-104050: Add typing to Argument Clinic converters (python#104547) pythonGH-103906: Remove immortal refcounting in the interpreter (pythonGH-103909) pythongh-87474: Fix file descriptor leaks in subprocess.Popen (python#96351) pythonGH-103092: isolate `pyexpat` (python#104506) pythongh-75367: Fix data descriptor detection in inspect.getattr_static (python#104517) pythongh-104050: Add more annotations to `Tools/clinic.py` (python#104544) pythongh-104555: Fix isinstance() and issubclass() for runtime-checkable protocols that use PEP 695 (python#104556) pythongh-103865: add monitoring support to LOAD_SUPER_ATTR (python#103866) CODEOWNERS: Assign new PEP 695 files to myself (python#104551) pythonGH-104510: Fix refleaks in `_io` base types (python#104516) pythongh-104539: Fix indentation error in logging.config.rst (python#104545) pythongh-104050: Don't star-import 'types' in Argument Clinic (python#104543) pythongh-104050: Add basic typing to CConverter in clinic.py (python#104538) ...

brandtbucher added 4 commits April 26, 2023 16:24

Remove a bunch of refcounting overhead

1aaf52c

make regen-cases

446757a

blurb add

138184d

fixup

fd95518

brandtbucher added performance sprint interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Apr 26, 2023

brandtbucher requested a review from hauntsaninja April 26, 2023 23:37

brandtbucher self-assigned this Apr 26, 2023

bedevere-bot mentioned this pull request Apr 26, 2023

Avoid reference counting operations on None, True, False in the interpreter #103906

Closed

bedevere-bot added the awaiting core review label Apr 26, 2023

brandtbucher mentioned this pull request Apr 26, 2023

gh-103906: Avoid refcount changes for True, False, None #103907

Closed

hauntsaninja approved these changes Apr 27, 2023

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Apr 27, 2023

sunmy2019 reviewed Apr 27, 2023

View reviewed changes

brandtbucher marked this pull request as ready for review April 27, 2023 04:15

brandtbucher added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 27, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 27, 2023

sobolevn reviewed Apr 27, 2023

View reviewed changes

Merge branch 'main' into immortal-interpreter

7f90633

markshannon approved these changes May 12, 2023

View reviewed changes

brandtbucher added 2 commits May 16, 2023 13:47

Catch up with main

733cdb4

One more...

56ec571

brandtbucher merged commit b4a9747 into python:main May 16, 2023

bedevere-bot removed the awaiting merge label May 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-103906: Remove immortal refcounting in the interpreter#103909

GH-103906: Remove immortal refcounting in the interpreter#103909
brandtbucher merged 7 commits into
python:mainfrom
brandtbucher:immortal-interpreter

brandtbucher commented Apr 26, 2023 •

edited by bedevere-bot

Loading

Uh oh!

hauntsaninja left a comment

Uh oh!

bedevere-bot commented Apr 27, 2023

Uh oh!

sobolevn left a comment

Uh oh!

JelleZijlstra commented Apr 27, 2023

Uh oh!

brandtbucher commented Apr 29, 2023

Uh oh!

markshannon left a comment

Uh oh!

JelleZijlstra commented May 12, 2023

Uh oh!

bedevere-bot commented May 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

brandtbucher commented Apr 26, 2023 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hauntsaninja left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Apr 27, 2023

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Apr 27, 2023

Uh oh!

brandtbucher commented Apr 29, 2023

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented May 12, 2023

Uh oh!

bedevere-bot commented May 16, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

brandtbucher commented Apr 26, 2023 •

edited by bedevere-bot

Loading