bpo-29347: Fix possibly dereferencing undefined pointers when creating weakref objects by zhangyangyu · Pull Request #128 · python/cpython

Repro for re._parser JIT correctness regression discovered via
pythia python#128 python#2 perf seam (generalist 22:05Z; supervisor-opened
W-RE-PARSER-DEOPT-DETACH workstream 22:06:56Z).

Workload: cinderjit.auto() + repeated re.compile() of named-group
patterns past the 1000-call auto-compile threshold so
re._parser:Tokenizer.__next + SubPattern.__getitem__ + .getwidth
all hit deopt-backoff + JIT-detach. Subsequent re.compile()
either:

  (a) AttributeError: 'SubPattern' object has no attribute 'state'
      at Lib/re/_compiler.py:761 (p.state.groupdict)
  (b) SIGSEGV (exit 139, core dumped)

Without JIT_ENABLE=1: 2000+ iterations PASS clean.

Two failure modes (AttributeError + SIGSEGV) on the same workload
suggest UAF / corrupted attribute storage post-deopt-detach rather
than a missing __init__ run (theologian 22:06:38Z structural read).

JIT trace messages preceding crash include sequential Deopt
backoff for Tokenizer.__next, Tokenizer.match, SubPattern.__getitem__,
SubPattern.getwidth (all reaching 1000 guard failures and
detaching), with Skipping compilation messages for SubPattern.__init__
+ State.__init__.

Test is currently @unittest.skip per W-A4-mid sentinel precedent
(test exists, currently skipped, will be un-skipped when the
underlying bug is fixed and the workload runs clean — at that
point this becomes a regression gate).

cinderx_dev oracle deferred until devgpu004 SSH-2FA infra restored;
per Alex policy "all bugs presumed Phoenix-introduced" the working
hypothesis is Phoenix regression, not pre-existing CinderX class.

Auth: pythia python#128 python#2 surfacing; generalist 22:05:55Z bug
escalation; theologian 22:06:38Z + supervisor 22:06:56Z CONCUR (α)
+ sentinel commit + W-RE-PARSER-DEOPT-DETACH workstream open.

…ntinel

Per docs/w-i3-runtime-assert-spec.md (theologian 21:57:02Z, supervisor
22:10:02Z, librarian 22:05:19Z framework guidance). Closes pythia
python#128 / python#131 python#1 named risk: I3 invariant (BasicBlock::id is allocation-
monotonic AND never mutated post-allocation) is grep-only, not
runtime-asserted.

Phase B B-gamma's PhxBcBlockArray (Tier 8 SECOND-PILOT, push 40
4145fe3) indexes by BasicBlock::id. A future HIR pass that
renumbers ids (SSA-destruction-style cache-locality renumbering,
peephole block-splitter, speculative-inlining clone) would silently
corrupt the dense-array lookup; I2 read-site check catches out-of-
range, NOT stale-mapping-correct-range.

(IV) CI grep gate — scripts/check_i3_invariant.sh + Step 1h wire:
- Detects "->id =", ".id =", set_id(/setId( patterns in Python/jit/hir/.
- Allow-list excludes legitimate non-BasicBlock matches:
  * HirLoadAttrSpecial::id (void* attribute identifier, different
    class)
  * next_register_id_, cache_id_ (different fields, trailing
    underscore)
  * test_*.c, hir_instr_c_verify.cpp (read-path testing, per W44
    ALLOW_LIST precedent)
- Falsification-tested: synthetic patch adding "block->id = 42;"
  to Python/jit/hir/synth_i3_violation.cpp triggers GATE FAIL EXIT=1
  under --strict; removing the patch returns to GATE PASS EXIT=0.
- Wired as Step 1h in scripts/gate_phoenix.sh per librarian 22:05:19Z
  framework guidance, mirroring Steps 1e/1g delegation shape (capture
  OUTPUT, tee to RESULTS_FILE, GATE_PASS=0 + FAILURES on nonzero).

(III) JIT_DCHECK pydebug sentinel — PhxBcBlockEntry.sentinel_id:
- Adds int sentinel_id field under #ifdef Py_DEBUG to PhxBcBlockEntry
  (zero release-build cost; pydebug entry size 8 -> 12 bytes).
- Insert sets sentinel_id = block_id at insert time.
- Lookup verifies sentinel_id == block_id via JIT_DCHECK_C; mismatch
  fires JIT_CHECK_C abort with diagnostic indicating either a
  post-allocation id mutation OR inserter/reader index disagreement.
- catches the (IV) miss-class: id mutation via member function /
  indirect pointer write that grep doesn't pattern-match.

Combined cost: zero release runtime; ~45min implementation. (I)
sentinel-always and (II) generation-counter REJECTED per spec §3 as
reverse perf trade without strong evidence the always-on detection
is required vs (IV)+(III) catching at commit + pydebug.

Build implications: Py_DEBUG-conditional layout change to
PhxBcBlockEntry. Per JIT Header Change Protocol (CLAUDE.md), requires
full distclean rebuild; testkeeper to verify both release (sentinel
field absent, struct == 8 bytes) and pydebug (sentinel field present,
struct == 12 bytes, JIT_DCHECK_C does not fire on normal workload).
ARM64 verification via standard ARM64 retroactive-debt path once
devgpu004 SSH-2FA infra restored.

Auth: theologian spec 21:57:02Z + supervisor CONCUR 21:57:20Z +
22:10:02Z; librarian framework guidance 22:05:19Z (Step 1h shape
mirroring 1e/1g, scripts/check_i3_invariant.sh --strict).

Per theologian W27c pair-batch design 00:08:52Z + 16:57:48Z
APPROVE; supervisor (α) local-accumulate posture. Conversion #2b
in the 5-method PARTIAL backlog (now 3 → 2 PARTIAL Cat-B
remaining); pair-batch sibling of W27c #2a (7135d94).

Cat B residue elimination — pair-batch closure: this commit
mirrors W27c #2a emitInlineExceptionMatch's shape and reuses the
same C helper build_inline_except_opcode_array_c (introduced in
#2a). The pre-resolve loop that previously ran on the C++ side
(BytecodeInstruction iteration + PyTuple_GET_ITEM(co_consts) for
LOAD_CONST/RETURN_CONST + getBlockAtOff for JUMP_BACKWARD* + the
terminator-detection skip-list) is deleted C++-side and replaced
with a single helper invocation in the C body.

Signature change: hir_builder_emit_call_exception_handler_c drops
(opcodes, opcode_count) params; builds opcode array internally
from PyCodeObject (via tc->frame.code) + builder block_map +
except_body_offset, identical to the #2a entry path. C++ stub
shrinks from ~70 lines to ~25.

D1 pre-amble (call_instr->setSuppressExceptionDeopt(true) +
phx_ptr_arr_pop on tc.frame.stack) stays C++-side because
DeoptBase*/Register* are C++-only types — that boundary is
intentional and unchanged.

D3 ok-block sequence (RefineType in-place SSA-rename + push
result back to outer tc stack to undo emitAnyCall's pre-amble
pop) preserved verbatim.

Bridge-struct elimination (bonus cleanup): with both #2a and #2b
now C-side for opcode-array construction, the C++/C bridge struct
OpcodeArrayEntry_CXX (+ its static_assert) in builder.cpp has
zero remaining references. Deleted entirely. The C-side
OpcodeArrayEntry struct in builder_emit_c.c stays (still used by
emit_except_match_body_c). This realises the design intent stated
in W27c #2a's commit message ("Eliminates OpcodeArrayEntry C++/C
bridge struct entirely").

Honest ratio framing per librarian D-1776879377 / D-1776880214:
post-conversion 98/100 PURE-CONVERTED + 2 PARTIAL Cat-B remaining
(was 97/100 + 3 PARTIAL after W27c #2a). emitCallExceptionHandler
is now a delegation stub, not pure-C++-eliminated; same metric
reading as the established honest-scope record. Remaining 2
PARTIAL are emitAnyCall (await-tail residue, design-residual per
pythia python#128 D-1777029012) and emitLoadMethodStatic (classloader.h
_PyType_VTable struct dependency, design-residual per A1 rule).

VERIFICATION: deferred to testkeeper full gate cycle (post-reboot
fresh verifications per supervisor 16:54:08Z stale-pre-reboot
discipline). All pre-reboot results (testkeeper 00:48:33Z +
01:45:23Z + 05:39:16Z) RETIRED. Push 54 batch (W27c python#1
7bfb7d1 + #2a 7135d94 + this commit) gated on:
- x86_64 fresh build + JIT smoke + nbody + W-RE-PARSER
  substrate-stability + Phoenix CPython suite + ABBA 4-bench
- ARM64 STRICT verify on 3-commit bundle (devgpu004 connectivity
  restored 16:53Z post-server-reboot)
- gatekeeper APPROVE
- supervisor PUSH AUTHORIZED

Auth chain: theologian design 00:08:52Z + APPROVE 16:57:48Z;
supervisor (α) authorization 00:07:10Z + 16:55:47Z plan; testkeeper
standby for fresh full gate (post-reboot discipline).

Per theologian W27c pair-batch design 00:08:52Z + 16:57:48Z
APPROVE; supervisor (α) local-accumulate posture. Conversion #2b
in the 5-method PARTIAL backlog (now 3 → 2 PARTIAL Cat-B
remaining); pair-batch sibling of W27c #2a (7135d94).

Cat B residue elimination — pair-batch closure: this commit
mirrors W27c #2a emitInlineExceptionMatch's shape and reuses the
same C helper build_inline_except_opcode_array_c (introduced in
#2a). The pre-resolve loop that previously ran on the C++ side
(BytecodeInstruction iteration + PyTuple_GET_ITEM(co_consts) for
LOAD_CONST/RETURN_CONST + getBlockAtOff for JUMP_BACKWARD* + the
terminator-detection skip-list) is deleted C++-side and replaced
with a single helper invocation in the C body.

Signature change: hir_builder_emit_call_exception_handler_c drops
(opcodes, opcode_count) params; builds opcode array internally
from PyCodeObject (via tc->frame.code) + builder block_map +
except_body_offset, identical to the #2a entry path. C++ stub
shrinks from ~70 lines to ~25.

D1 pre-amble (call_instr->setSuppressExceptionDeopt(true) +
phx_ptr_arr_pop on tc.frame.stack) stays C++-side because
DeoptBase*/Register* are C++-only types — that boundary is
intentional and unchanged.

D3 ok-block sequence (RefineType in-place SSA-rename + push
result back to outer tc stack to undo emitAnyCall's pre-amble
pop) preserved verbatim.

Bridge-struct elimination (bonus cleanup): with both #2a and #2b
now C-side for opcode-array construction, the C++/C bridge struct
OpcodeArrayEntry_CXX (+ its static_assert) in builder.cpp has
zero remaining references. Deleted entirely. The C-side
OpcodeArrayEntry struct in builder_emit_c.c stays (still used by
emit_except_match_body_c). This realises the design intent stated
in W27c #2a's commit message ("Eliminates OpcodeArrayEntry C++/C
bridge struct entirely").

Honest ratio framing per librarian D-1776879377 / D-1776880214:
post-conversion 98/100 PURE-CONVERTED + 2 PARTIAL Cat-B remaining
(was 97/100 + 3 PARTIAL after W27c #2a). emitCallExceptionHandler
is now a delegation stub, not pure-C++-eliminated; same metric
reading as the established honest-scope record. Remaining 2
PARTIAL are emitAnyCall (await-tail residue, design-residual per
pythia python#128 D-1777029012) and emitLoadMethodStatic (classloader.h
_PyType_VTable struct dependency, design-residual per A1 rule).

VERIFICATION: deferred to testkeeper full gate cycle (post-reboot
fresh verifications per supervisor 16:54:08Z stale-pre-reboot
discipline). All pre-reboot results (testkeeper 00:48:33Z +
01:45:23Z + 05:39:16Z) RETIRED. Push 54 batch (W27c python#1
7bfb7d1 + #2a 7135d94 + this commit) gated on:
- x86_64 fresh build + JIT smoke + nbody + W-RE-PARSER
  substrate-stability + Phoenix CPython suite + ABBA 4-bench
- ARM64 STRICT verify on 3-commit bundle (devgpu004 connectivity
  restored 16:53Z post-server-reboot)
- gatekeeper APPROVE
- supervisor PUSH AUTHORIZED

Auth chain: theologian design 00:08:52Z + APPROVE 16:57:48Z;
supervisor (α) authorization 00:07:10Z + 16:55:47Z plan; testkeeper
standby for fresh full gate (post-reboot discipline).