Remove useless &PyRef patterns by youknowone · Pull Request #6455 · RustPython/RustPython
Walkthrough
This PR shifts many internal APIs to use owned/wrapper types (Py<T>, PyObject) instead of borrowed refs (PyRef<T>, PyObjectRef, PyBaseExceptionRef, etc.), updating signatures and call sites across stdlib, VM builtins, ctypes, exception handling, and related modules.
Changes
| Cohort / File(s) | Summary |
|---|---|
CSV module crates/stdlib/src/csv.rs |
Parsing helpers changed from &PyObjectRef → &PyObject and use .to_owned(); error messages refined for quote/escape/lineterminator/quoting parsing. |
Exception handling / SSL / WASM crates/stdlib/src/openssl.rs, crates/stdlib/src/ssl/compat.rs, crates/wasm/src/convert.rs, crates/vm/src/exceptions.rs, crates/vm/src/coroutine.rs, crates/vm/src/vm/mod.rs, crates/vm/src/import.rs |
Many functions changed from PyBaseExceptionRef → Py<PyBaseException> (signatures updated for write_exception, contextualize/set context, is_gen_exit, remove_importlib_frames, py_err_to_js_err, etc.). Small import/export ordering edits. |
scproxy / dict view / JIT crates/stdlib/src/scproxy.rs, crates/vm/src/builtins/dict.rs, crates/vm/src/builtins/function/jit.rs |
Dict types moved from PyDictRef → Py<PyDict>: closure signatures, DictView::dict() trait, macro impls, JIT arg lookup updated; mapping proxy construction uses .to_owned(). |
Sequence & collection element access crates/vm/src/builtins/list.rs, crates/vm/src/stdlib/collections.rs, crates/vm/src/sequence.rs |
MutObjectSequenceOp::do_get return type changed from Option<&PyObjectRef> → Option<&PyObject>; callers use `.map( |
Tuple/tuple-like & codecs crates/vm/src/builtins/genericalias.rs, crates/vm/src/codecs.rs |
Helpers (tuple_index, unpack detection) now accept &PyObject; subs_tvars and tuple-returning APIs moved from &PyTupleRef → &Py<PyTuple>; PyCodec::as_tuple() signature updated. |
Type / payload / ConstName crates/vm/src/builtins/type.rs, crates/vm/src/object/payload.rs, crates/vm/src/vm/context.rs |
new_simple_heap and payload error helpers now take &Py<PyType> (instead of PyTypeRef); ConstName::new accepts &Py<PyType> and uses to_owned(). |
Property / builtin function crates/vm/src/builtins/property.rs, crates/vm/src/builtins/builtin_func.rs |
Internal closures and helpers adjusted from &PyObjectRef → &PyObject; PyNativeFunction::get_self() return type changed to Option<&PyObject> and is non-const. |
ctypes (multiple files) crates/vm/src/stdlib/ctypes.rs, crates/vm/src/stdlib/ctypes/base.rs, crates/vm/src/stdlib/ctypes/field.rs, crates/vm/src/stdlib/ctypes/structure.rs, crates/vm/src/stdlib/ctypes/union.rs |
APIs updated to accept PyObject in many helpers (e.g., new_simple_type uses Either<&PyObject, &PyTypeRef>; get_field_size/get_field_align and value_to_bytes use &PyObject); ownership handling standardized via .to_owned(). |
Exception group / derive logic crates/vm/src/exception_group.rs |
Multiple helpers converted to PyObject / Py<PyBaseException> signatures; get_condition_matcher and derive_and_copy_attributes expanded for tuple/type/Callable handling and attribute propagation. |
Formatting / scope / suggestions / warnings / frame crates/vm/src/cformat.rs, crates/vm/src/stdlib/builtins.rs, crates/vm/src/stdlib/typevar.rs, crates/vm/src/suggestion.rs, crates/vm/src/warn.rs, crates/vm/src/frame.rs |
Numerous helper params switched from PyObjectRef → PyObject and some exception refs to Py<PyBaseException>; set_module_from_caller and related helpers updated; small comment prototype change in scope. |
Misc small API adjustments crates/vm/src/stdlib/ctypes/field.rs, crates/vm/src/stdlib/ctypes/structure.rs, crates/stdlib/src/ssl.rs, crates/wasm/src/js_module.rs |
Local helpers updated to new owned/wrapper types; some .clone() → .to_owned() conversions and minor import edits. |
Estimated code review effort
🎯 4 (Complex) | ⏱️ ~60 minutes
- Pay particular attention to:
- exception group derivation and attribute-copy semantics (
crates/vm/src/exception_group.rs) - ctypes field size/align and type-discovery fallbacks (
crates/vm/src/stdlib/ctypes/*) - propagation of
DictView::dict()change across all impls/consumers (crates/vm/src/builtins/dict.rs) - places switching from borrowed refs to
Py<...>for lifetime/ownership correctness in VM exception paths
- exception group derivation and attribute-copy semantics (
Possibly related PRs
- native ExceptionGroup #6358 — overlaps changes in
exception_group.rs(ConditionMatcher, derive_and_copy_attributes). - ctypes buffer #6311 — extensive ctypes refactors touching the same
ctypesfiles and ownership patterns. - Fix misused PyTypeError #6412 — related edits to CSV quoting/QuoteStyle parsing and error handling.
Suggested reviewers
- ShaharNaveh
- arihant2math
Poem
🐰
I nibbled refs and found them light,
Swapped clones for to_owned — what a sight!
Py snug in a burrow neat,
PyObject hops on nimble feet.
Cheers — the VM's now tidy and bright!
Pre-merge checks and finishing touches
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 77.53% which is insufficient. The required threshold is 80.00%. | You can run @coderabbitai generate docstrings to improve docstring coverage. |
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The title accurately describes the main objective of this PR: systematically replacing PyRef and PyObjectRef patterns with Py and PyObject types throughout the codebase. |
✨ Finishing touches
- 📝 Generate docstrings
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Post copyable unit tests in a comment
📜 Recent review details
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (17)
crates/stdlib/src/openssl.rs(2 hunks)crates/stdlib/src/scproxy.rs(2 hunks)crates/stdlib/src/ssl/compat.rs(3 hunks)crates/vm/src/builtins/dict.rs(5 hunks)crates/vm/src/builtins/function/jit.rs(2 hunks)crates/vm/src/builtins/genericalias.rs(2 hunks)crates/vm/src/builtins/union.rs(1 hunks)crates/vm/src/codecs.rs(1 hunks)crates/vm/src/coroutine.rs(2 hunks)crates/vm/src/exception_group.rs(3 hunks)crates/vm/src/exceptions.rs(6 hunks)crates/vm/src/import.rs(2 hunks)crates/vm/src/scope.rs(1 hunks)crates/vm/src/suggestion.rs(3 hunks)crates/vm/src/vm/mod.rs(3 hunks)crates/wasm/src/convert.rs(2 hunks)crates/wasm/src/js_module.rs(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (7)
- crates/stdlib/src/openssl.rs
- crates/vm/src/vm/mod.rs
- crates/wasm/src/convert.rs
- crates/vm/src/builtins/genericalias.rs
- crates/vm/src/suggestion.rs
- crates/stdlib/src/scproxy.rs
- crates/vm/src/scope.rs
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.rs: Follow the default rustfmt code style by runningcargo fmtto format Rust code
Always run clippy to lint Rust code (cargo clippy) before completing tasks and fix any warnings or lints introduced by changes
Follow Rust best practices for error handling and memory management
Use the macro system (pyclass,pymodule,pyfunction, etc.) when implementing Python functionality in Rust
Files:
crates/wasm/src/js_module.rscrates/vm/src/codecs.rscrates/vm/src/coroutine.rscrates/vm/src/exception_group.rscrates/vm/src/import.rscrates/stdlib/src/ssl/compat.rscrates/vm/src/builtins/function/jit.rscrates/vm/src/builtins/dict.rscrates/vm/src/exceptions.rscrates/vm/src/builtins/union.rs
🧠 Learnings (4)
📚 Learning: 2025-12-09T08:46:58.660Z
Learnt from: youknowone
Repo: RustPython/RustPython PR: 6358
File: crates/vm/src/exception_group.rs:173-185
Timestamp: 2025-12-09T08:46:58.660Z
Learning: In crates/vm/src/exception_group.rs, the derive() method intentionally always creates a BaseExceptionGroup instance rather than preserving the original exception class type. This is a deliberate design decision that differs from CPython's behavior.
Applied to files:
crates/wasm/src/js_module.rscrates/vm/src/coroutine.rscrates/vm/src/exception_group.rscrates/vm/src/import.rscrates/vm/src/exceptions.rs
📚 Learning: 2025-11-10T06:27:41.954Z
Learnt from: youknowone
Repo: RustPython/RustPython PR: 6243
File: vm/src/function/buffer.rs:123-129
Timestamp: 2025-11-10T06:27:41.954Z
Learning: In vm/src/function/buffer.rs, line 36 in the try_rw_bytes_like method intentionally uses TypeError (not BufferError) for the "buffer is not a read-write bytes-like object" error case, even though a similar error message in ArgMemoryBuffer::try_from_borrowed_object uses BufferError. The TypeError is the intended exception type for this specific code path.
Applied to files:
crates/wasm/src/js_module.rs
📚 Learning: 2025-11-29T12:17:28.606Z
Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-29T12:17:28.606Z
Learning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files in the `Lib/` directory; modifications should be minimal and only to work around RustPython limitations
Applied to files:
crates/vm/src/import.rscrates/stdlib/src/ssl/compat.rs
📚 Learning: 2025-11-29T12:17:28.606Z
Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-29T12:17:28.606Z
Learning: Applies to **/*.rs : Use the macro system (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust
Applied to files:
crates/vm/src/import.rs
🧬 Code graph analysis (3)
crates/wasm/src/js_module.rs (1)
crates/vm/src/builtins/type.rs (1)
new_simple_heap(176-189)
crates/vm/src/exception_group.rs (4)
crates/vm/src/exceptions.rs (4)
obj(221-221)obj(225-225)obj(395-395)obj(403-403)crates/vm/src/stdlib/ctypes/function.rs (3)
typ(34-35)t(86-86)t(364-364)crates/vm/src/stdlib/ctypes/array.rs (4)
typ(80-85)typ(464-466)t(82-82)t(90-90)crates/vm/src/frame.rs (1)
func(2121-2122)
crates/vm/src/builtins/function/jit.rs (1)
crates/vm/src/builtins/dict.rs (1)
dict(756-756)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: Run snippets and cpython tests on wasm-wasi
- GitHub Check: Ensure compilation on various targets
- GitHub Check: Run rust tests (windows-latest)
- GitHub Check: Run rust tests (macos-latest)
- GitHub Check: Run snippets and cpython tests (windows-latest)
- GitHub Check: Check the WASM package and demo
- GitHub Check: Check Rust code with clippy
- GitHub Check: Run snippets and cpython tests (ubuntu-latest)
- GitHub Check: Run rust tests (ubuntu-latest)
- GitHub Check: Run snippets and cpython tests (macos-latest)
- GitHub Check: Run tests under miri
🔇 Additional comments (23)
crates/stdlib/src/ssl/compat.rs (3)
26-29: LGTM! Import updates align with the signature changes below.The addition of
PyBaseExceptionandPyimports properly supports the type signature changes from&PyBaseExceptionRefto&Py<PyBaseException>.
987-989: LGTM! Correct simplification of reference type.Changing from
&PyBaseExceptionRef(i.e.,&PyRef<PyBaseException>) to&Py<PyBaseException>removes unnecessary indirection sincePyRef<T>derefs toPy<T>. The function body usingfast_isinstanceremains valid as it's available onPy<T>.
1537-1556: LGTM! Consistent with the pattern applied throughout this PR.The signature change to
&Py<PyBaseException>is correct. All method calls (fast_isinstance,as_object,get_attr,try_int,try_to_primitive) are properly available onPy<PyBaseException>.crates/vm/src/import.rs (2)
3-10: LGTM! Import reorganization is clean and correct.The imports are properly updated to include
PyandPyBaseExceptionfromexceptions::types, supporting the signature change inremove_importlib_frames.
208-218: LGTM! Signature change is correct and body remains valid.The change from
&PyBaseExceptionRefto&Py<PyBaseException>properly simplifies the reference type. All method calls (fast_isinstance,__traceback__,set_traceback_typed) work correctly onPy<PyBaseException>.crates/vm/src/exceptions.rs (6)
79-86: LGTM! Entry point for exception writing correctly updated.The signature change to
&Py<PyBaseException>is appropriate and cascades consistently to the internal helper methods.
88-129: LGTM! Recursive exception writing with proper type handling.The parameter change is correct. The
get_id()and recursive calls work properly with&Py<PyBaseException>. The&cause_or_contextreference on line 121 correctly passes a reference to the innerPy<PyBaseException>.
132-170: LGTM! Inner exception writing logic remains valid.All method accesses (
.traceback.read(),.args(),.class(),.as_object()) are properly available onPy<PyBaseException>.
177-183: LGTM! SyntaxError formatting with updated types.Both
exc: &Py<PyBaseException>andexc_type: &Py<PyType>parameter types are correct. The&Py<PyType>change aligns with the broader pattern of this PR.
370-385: LGTM! Traceback entry writing correctly uses&Py<PyTraceback>.The change from
&PyTracebackRefto&Py<PyTraceback>is consistent with the PR pattern. Field accesses (.frame,.lineno) work correctly on thePy<PyTraceback>type.
1053-1064: LGTM! SerializeException struct and constructor updated consistently.The struct field and
new()parameter now use&'s Py<PyBaseException>instead of&'s PyBaseExceptionRef. This removes unnecessary indirection while maintaining the same functionality. TheSerializeExceptionOwnedvariant (line 1067-1070) correctly retainsPyBaseExceptionRefsince it owns the exception rather than borrowing it.crates/wasm/src/js_module.rs (1)
615-615: LGTM! Type signature change aligns with PR objective.The type passing for
vm.ctx.exceptions.exception_typeis correct—it's a&'static Py<PyType>that satisfies the&Py<PyType>parameter ofnew_simple_heap. The Rust compiler enforces type correctness at compile time, and this change maintains type safety.crates/vm/src/coroutine.rs (2)
2-2: LGTM! Imports correctly support the type system change.The addition of
PyandPyBaseExceptionimports aligns with the shift to owned/wrapper types and supports the updatedis_gen_exitsignature.Also applies to: 5-5
211-213: Type mismatch: function signature incompatible with current call site.The signature change to
&Py<PyBaseException>introduces a compilation error. At line 168, the error typeefromPyResultisPyBaseExceptionRef(alias forPyRef<PyBaseException>), notPy<PyBaseException>. These are incompatible types. Either updatePyResultto usePy<PyBaseException>as its error type, or revert the function signature to accept&PyBaseExceptionRef.⛔ Skipped due to learnings
Learnt from: youknowone Repo: RustPython/RustPython PR: 6243 File: vm/src/function/buffer.rs:123-129 Timestamp: 2025-11-10T06:27:41.954Z Learning: In vm/src/function/buffer.rs, line 36 in the try_rw_bytes_like method intentionally uses TypeError (not BufferError) for the "buffer is not a read-write bytes-like object" error case, even though a similar error message in ArgMemoryBuffer::try_from_borrowed_object uses BufferError. The TypeError is the intended exception type for this specific code path.crates/vm/src/exception_group.rs (3)
355-429: LGTM! Clean type refactoring.The signature changes from
&PyObjectRef/&PyRef<T>to&PyObject/&Py<T>are consistent with the PR's goal. The shift fromclone()toto_owned()at lines 412 and 424 correctly expresses the intent of converting borrowed references to owned references. Alldowncast_refusages remain appropriate for borrowed types.
431-469: LGTM! Signature change preserves all functionality.The parameter type change from
&PyRef<PyBaseException>to&Py<PyBaseException>aligns with the PR objective. All attribute access patterns (__traceback__(),__context__(),__cause__(),as_object()) work correctly with the new type, and the metadata propagation logic remains intact.
355-469: Runcargo clippyto verify no lints are introduced.Per the coding guidelines, clippy must be run on all Rust changes before completion. While the code appears structurally sound with proper error handling and memory management, the clippy check must be executed to ensure no warnings are introduced by these helper functions.
crates/vm/src/builtins/function/jit.rs (2)
3-5: LGTM! Import addition aligns with signature refactor.The addition of
PyDictto the imports is necessary for the updated function signature on line 47. KeepingPyDictRefensures compatibility with existing call sites.
47-47: Type coercion from&PyDictRefto&Py<PyDict>is correct. The signature change is valid becausePyDictRefis a type alias forPyRef<PyDict>, andPyRef<T>implementsDerefwithTarget = Py<T>. This allows&dict(wheredictisPyDictRef) to automatically coerce to&Py<PyDict>at call sites (lines 98, 102, 121) via Rust's deref coercion mechanism. The pattern is sound and type-safe.crates/vm/src/builtins/dict.rs (2)
756-756: LGTM! Trait signature updated consistently.The return type change from
&PyDictRefto&Py<PyDict>in theDictViewtrait aligns with the PR's objective to shift from borrowed refs to owned/wrapper types. The implementation in thedict_view!macro (line 788) correctly reflects this change.
1145-1145: No action needed. The type compatibility is correct.The change from
.clone()to.to_owned()is valid. When.to_owned()is called on&Py<PyDict>, it returnsPyRef<PyDict>(aliasPyDictRef) via theimpl<T> ToOwned for Py<T>trait, which directly matches the existingimpl From<PyDictRef> for PyMappingProxy. No type mismatch exists.crates/vm/src/builtins/union.rs (1)
45-47: Review comment is based on incorrect assumptions and should be revised.The method signature does not involve a
constremoval—noconstqualifier appears on this method. The return type works through automaticDerefcoercion:PyRef<T>implementsDerefwith targetPy<T>, so&self.args(which is&PyRef<PyTuple>) automatically coerces to&Py<PyTuple>. This is a transparent implementation mechanism, not a breaking API change requiring downstream verification. The call sites in the codebase are already compatible.Likely an incorrect or invalid review comment.
crates/vm/src/codecs.rs (1)
55-57: Methodas_tuple()has no usages in the codebase; verification request cannot be completed.The method returns
&Py<PyTuple>and is technically correct due to Deref coercion fromPyRef<PyTuple>, but it is dead code with zero call sites. The return type annotation returns a reference to an owned type, not an owned type itself, which conflicts with the stated goal of standardizing on ownedPy<T>types. This method should either be removed if unused or its purpose clarified if it's part of a planned public API.Likely an incorrect or invalid review comment.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.