GH-104142: Fix `_Py_RefcntAdd` to respect immortality by brandtbucher · Pull Request #104143 · python/cpython
🤖 New build scheduled with the buildbot fleet by @brandtbucher for commit be3b103 🤖
If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
(Not sure why buildbots aren't running...)
@brandtbucher Ah would you like to add the C API test in here too?
| static PyMethodDef test_methods[] = { | |
| {"test_immortal_builtins", test_immortal_builtins, METH_NOARGS}, | |
| {"test_immortal_small_ints", test_immortal_small_ints, METH_NOARGS}, | |
| {NULL}, | |
| }; |
Adding a code line somewhere of here would be enough.
| assert(_Py_IsImmortal(object)); | |
| Py_ssize_t old_count = Py_REFCNT(object); | |
| for (int j = 0; j < 10000; j++) { | |
| Py_DECREF(object); | |
| } | |
| Py_ssize_t current_count = Py_REFCNT(object); |
Comment on lines +61 to +63
| if (_Py_IsImmortal(op)) { | ||
| return; | ||
| } |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One quick comment here is that if this were to be in a hot path, I would recommend using the same technique that we use for Py_INCREF to minimize the number of generated instructions to do the check and add: https://github.com/python/cpython/blob/main/Include/object.h#L620-L634
Now, given that it's only in use for sq_repeat in tuple and list, I don't think this is perf sensitive so keeping the code as you have it here should be good.
| self.assertEqual(sys.getrefcount(immortal), self.IMMORTAL_REFCOUNT) | ||
|
|
||
| def test_immortals(self): | ||
| for immortal in self.IMMORTALS: |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small nit: I generally prefer rolling out assertions in unit tests to make it easier to isolate by line-number in the place where the test failed. But I don't think there's a standard in the code base so I'm fine either way
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'd like comprehensive testing of these immortals though, and I don't want to roll out ~250 different assertions for each test.
I figured making each one a subTest in assert_immortal is a pretty good compromise.
Nice catch, I could have sworn that I had added the check to _Py_RefcntAdd as well but I somehow missed it. I added two comments but they are minor so this LGTM.
Thanks for the fix!
LGTM, all increments to ob->ob_refcnt should be guarded now.
There is a USan failure:
Include/object.h:227:12: runtime error: member access within address 0x7fff7ae78b80 with insufficient space for an object of type 'PyObject' (aka 'struct _object')
0x7fff7ae78b80: note: pointer points here
b8 7f 00 00 00 47 a8 c2 97 55 00 00 00 b5 59 7e 78 cd 65 f5 00 8a ea c2 b8 7f 00 00 39 7a 90 c0
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/object.h:227:12 in
make: *** [Makefile:1106: checksharedmods] Error 1
Not sure what that means, but the line is in _Py_IsImmortal (https://github.com/brandtbucher/cpython/blob/py-refcnt-add-immortal/Include/object.h#L227), so it may be real.
🤖 New build scheduled with the buildbot fleet by @JelleZijlstra for commit 1471b39 🤖
If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
(Trying again just in case it was something flaky.)
I honestly don't know what to do here.
It feels like a bug in USan, since I honestly don't get how the new code could cause that failure. But it's reproducible, and indeed seems to indeed be related to my change here.
So it's an existing issue (but one that probably should still be looked into).
See #104190
