Align marshal and .pyc with CPython 3.14 by youknowone · Pull Request #7958 · RustPython/RustPython

youknowone

added 3 commits

May 23, 2026 20:16

read_marshal_bytes, _str, _str_vec, _name_tuple, and _const_tuple now
take a shared ref table and resolve TYPE_REF / register FLAG_REF
entries. deserialize_code is split into a public wrapper and an inner
function that receives the ref table; deserialize_value_depth opens a
fresh inner ref space when it hits Type::Code, mirroring CPython's
behaviour of putting the code object itself at ref slot 0. Nested code
objects inside const tuples reuse the surrounding code's ref space via
the new read_const_value helper.

… 3.14

PYC_MAGIC_NUMBER changes from 2994 to 3627, matching CPython 3.14's
pyc_magic_number_token (0x0a0d0e2b). marshal FORMAT_VERSION drops from
5 to 4 (the encoder/marshal.version value; the decoder already accepts
both). check_pyc_magic_number_bytes now compares all four magic bytes
instead of the first two.

Two fixture-based tests pin the marshal decoder against actual CPython
3.14 marshal.dumps() output: a trivial module that exercises FLAG_REF
plus TYPE_REF for qualname, and a module with a nested function that
exercises ref sharing between a const tuple and its surrounding code
object.

SourceFileLoader.get_code now also looks for .pyc files using
_RP_FALLBACK_CACHE_TAGS (currently ('cpython-314',)) in addition to
sys.implementation.cache_tag. The matched .pyc is only used for
reading; recompilation still writes to the RustPython-tagged path, so
CPython's .pyc is never overwritten. Source-stat / hash / timestamp
validation logic is unchanged.

CPython's marshal supports TYPE_SLICE from format version 4 onwards
and that is the default version. Rejecting slice dumps below version
5 made marshal.dumps(slice(...)) fail with the default version and
broke test.test_marshal.SliceTestCase.test_slice.

Lib/importlib/_bootstrap_external.py is CPython's own code copied
verbatim; local patches here defeat compatibility tracking. The
cpython-XX cache_tag fallback needs to live on the RustPython side
(Rust code or sys.implementation.cache_tag policy), not as edits to
the imported standard library.

This reverts commit 1fc426d0fb5fcdb50d35cad13bbb43e8f6ce1c7f.

youknowone changed the title ~~Align marshal~~ Align marshal and .pyc with CPython 3.14

May 24, 2026

Use the CPython compatibility version (e.g. cpython-314) instead of
the rustpython-{MAJOR_IMPL}_{MINOR_IMPL} interpreter version string.

Py_MARSHAL_VERSION is 5 in CPython 3.14.5 (Include/marshal.h:16) and
TYPE_SLICE serialization rejects version < 5 (Python/marshal.c:720).
Restore the same threshold and constant so marshal.version and the
slice-marshal gate match CPython.

Code objects embedded in const-tuples reset the depth budget on each
recursion, so a hostile or pathological marshal stream of code-in-tuple-
in-code can blow the stack despite MAX_MARSHAL_STACK_DEPTH. Pass the
current depth through deserialize_code_inner and read_marshal_const_tuple
and decrement at each code-object/tuple boundary.

Also route dict keys through deserialize_value_after_header so TYPE_CODE
keys decode instead of failing with BadType.