◐ Shell
reader mode source ↗
Skip to content

GH-104142: Fix _Py_RefcntAdd to respect immortality#104143

Merged
brandtbucher merged 4 commits into
python:mainfrom
brandtbucher:py-refcnt-add-immortal
May 5, 2023
Merged

GH-104142: Fix _Py_RefcntAdd to respect immortality#104143
brandtbucher merged 4 commits into
python:mainfrom
brandtbucher:py-refcnt-add-immortal

Conversation

@brandtbucher

@brandtbucher brandtbucher commented May 3, 2023 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

Also, make the existing immortality tests a bit stricter and more comprehensive.

@brandtbucher brandtbucher added interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker labels May 3, 2023
@brandtbucher brandtbucher self-assigned this May 3, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 3, 2023
Closed
@brandtbucher brandtbucher added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2023
@bedevere-bot

bedevere-bot commented May 3, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @brandtbucher for commit be3b103 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2023
@ericsnowcurrently

ericsnowcurrently commented May 4, 2023

Copy link
Copy Markdown
Member

CC @eduardo-elizondo

@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

(Not sure why buildbots aren't running...)

corona10
corona10 previously approved these changes May 4, 2023
View reviewed changes
@corona10

corona10 commented May 4, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

@brandtbucher Ah would you like to add the C API test in here too?

cpython/Modules/_testcapi/immortal.c

Lines 34 to 38 in 9885677

static PyMethodDef test_methods[] = {
{"test_immortal_builtins", test_immortal_builtins, METH_NOARGS},
{"test_immortal_small_ints", test_immortal_small_ints, METH_NOARGS},
{NULL},
};

Adding a code line somewhere of here would be enough.

cpython/Modules/_testcapi/immortal.c

Lines 5 to 10 in 9885677

assert(_Py_IsImmortal(object));
Py_ssize_t old_count = Py_REFCNT(object);
for (int j = 0; j < 10000; j++) {
Py_DECREF(object);
}
Py_ssize_t current_count = Py_REFCNT(object);

@corona10

corona10 commented May 4, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Ah sorry please ignore #104143 (comment)
it's not public API...

@eduardo-elizondo

eduardo-elizondo commented May 4, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Nice catch, I could have sworn that I had added the check to _Py_RefcntAdd as well but I somehow missed it. I added two comments but they are minor so this LGTM.

Thanks for the fix!

@sunmy2019

sunmy2019 commented May 4, 2023

Copy link
Copy Markdown
Member

LGTM, all increments to ob->ob_refcnt should be guarded now.

@JelleZijlstra

JelleZijlstra commented May 4, 2023

Copy link
Copy Markdown
Member

There is a USan failure:

Include/object.h:227:12: runtime error: member access within address 0x7fff7ae78b80 with insufficient space for an object of type 'PyObject' (aka 'struct _object')
0x7fff7ae78b80: note: pointer points here
 b8 7f 00 00  00 47 a8 c2 97 55 00 00  00 b5 59 7e 78 cd 65 f5  00 8a ea c2 b8 7f 00 00  39 7a 90 c0
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/object.h:227:12 in 
make: *** [Makefile:1106: checksharedmods] Error 1

Not sure what that means, but the line is in _Py_IsImmortal (https://github.com/brandtbucher/cpython/blob/py-refcnt-add-immortal/Include/object.h#L227), so it may be real.

@JelleZijlstra JelleZijlstra added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 4, 2023
@bedevere-bot

bedevere-bot commented May 4, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @JelleZijlstra for commit 1471b39 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 4, 2023
@JelleZijlstra

JelleZijlstra commented May 4, 2023

Copy link
Copy Markdown
Member

(Trying again just in case it was something flaky.)

@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

Grrr... it failed again.

@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

I honestly don't know what to do here.

It feels like a bug in USan, since I honestly don't get how the new code could cause that failure. But it's reproducible, and indeed seems to indeed be related to my change here.

@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

Ah, wait, the same buildbot is also failing on main: https://buildbot.python.org/all/#/builders/719/builds/2611

So it's an existing issue (but one that probably should still be looked into).

@brandtbucher brandtbucher merged commit ce871fd into python:main May 5, 2023
@sunmy2019

sunmy2019 commented May 5, 2023

Copy link
Copy Markdown
Member

So it's an existing issue (but one that probably should still be looked into).

See #104190

@brandtbucher

brandtbucher commented May 5, 2023

Copy link
Copy Markdown
Member Author

Nice catch!

carljm added a commit to carljm/cpython that referenced this pull request May 5, 2023
@carljm
Merge branch 'main' into inlinecomp2 …
8a78a36 
* main: (61 commits)
  pythongh-64595: Argument Clinic: Touch source file if any output file changed (python#104152)
  pythongh-64631: Test exception messages in cloned Argument Clinic funcs (python#104167)
  pythongh-68395: Avoid naming conflicts by mangling variable names in Argument Clinic (python#104065)
  pythongh-64658: Expand Argument Clinic return converter docs (python#104175)
  pythonGH-103092: port `_asyncio` freelist to module state (python#104196)
  pythongh-104051: fix crash in test_xxtestfuzz with -We (python#104052)
  pythongh-104190: fix ubsan crash (python#104191)
  pythongh-104106: Add gcc fallback of mkfifoat/mknodat for macOS (pythongh-104129)
  pythonGH-104142: Fix _Py_RefcntAdd to respect immortality (pythonGH-104143)
  pythongh-104112: link from cached_property docs to method-caching FAQ (python#104113)
  pythongh-68968: Correcting message display issue with assertEqual (python#103937)
  pythonGH-103899: Provide a hint when accidentally calling a module (pythonGH-103900)
  pythongh-103963: fix 'make regen-opcode' in out-of-tree builds (python#104177)
  pythongh-102500: Add PEP 688 and 698 to the 3.12 release highlights (python#104174)
  pythonGH-81079: Add case_sensitive argument to `pathlib.Path.glob()` (pythonGH-102710)
  pythongh-91896: Deprecate collections.abc.ByteString (python#102096)
  pythongh-99593: Add tests for Unicode C API (part 2) (python#99868)
  pythongh-102500: Document PEP 688 (python#102571)
  pythongh-102500: Implement PEP 688 (python#102521)
  pythongh-96534: socketmodule: support FreeBSD divert(4) socket (python#96536)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10 corona10 corona10 approved these changes

@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently

+1 more reviewer

@eduardo-elizondo eduardo-elizondo eduardo-elizondo approved these changes

Assignees

@brandtbucher brandtbucher

Labels

interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker

Projects

Release and Deferred blockers 🚫
Status: Done +1 more

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@brandtbucher @bedevere-bot @ericsnowcurrently @corona10 @eduardo-elizondo @sunmy2019 @JelleZijlstra