ctypes overhaul by youknowone · Pull Request #6450 · RustPython/RustPython

youknowone

Walkthrough

Major multi-module refactor: spell/config updates; standardized dunder/default binding attributes; PEP3118-aware format parsing and buffer validation; Py upcast helpers; and a large, modular redesign of the _ctypes subsystem (new Initializer/SetAttr lifecycle, array/pointer/structure/simple/union, libffi call plumbing, visibility/signature changes).

Changes

Cohort / File(s)	Summary
Spell-check & config `\.cspell.dict/python-more.txt`, `\.cspell.json`	Updated cspell dictionary and config: added tokens (`needsfree`, `PYTHONAPI`, `oparg`, `pytype`, `sighandler`, etc.), replaced `pyarg`→`pyargs`, removed `unconstructible`.
Dunder/default binding renames `crates/vm/src/builtins/str.rs`, `crates/vm/src/builtins/function.rs`, `crates/vm/src/builtins/builtin_func.rs`, `crates/vm/src/builtins/object.rs`, `crates/vm/src/builtins/type.rs`, `crates/vm/src/exceptions.rs`, `crates/vm/src/stdlib/functools.rs`, `crates/vm/src/stdlib/ast/python.rs`	Simplified #[pymethod]/#[pygetset]/#[pyattr] usage and aligned Rust identifiers with Python-visible dunders/attributes (e.g., `is_ascii`→`isascii`, `get_class`→`__class__`, `get_mro`→`__mro__`, `get_suppress_context`→`__suppress_context__`, `fields`→`_fields`).
Buffer parsing, validation & helpers `crates/stdlib/src/pystruct.rs`, `crates/vm/src/buffer.rs`, `crates/vm/src/protocol/buffer.rs`, `crates/vm/src/types/structseq.rs`, `crates/vm/src/object/core.rs`	Replaced `is_ascii()`→`isascii()` calls; enhanced FormatCode::parse to skip/consume PEP3118 constructs (`T{...}`, `X{...}`, `(...)`, `:name:`) and report unmatched-construct errors; BufferDescriptor::validate now supports scalar ndim()==0; added `Py<T>::to_base` and `upcast_ref`.
_ctypes modularization & major rework `crates/vm/src/stdlib/ctypes.rs`, `crates/vm/src/stdlib/ctypes/*` (e.g. `array.rs`, `pointer.rs`, `function.rs`, `structure.rs`, `union.rs`, `simple.rs`, `library.rs`, `util.rs`, `thunk.rs`)	Large redesign: split module into submodules; new Initializer/SetAttr lifecycles; many types now wrap `PyCData`; added array-type caching, wchar support, pointer contents/indexing and slice semantics; expanded PyCFuncPtr/libffi conversion and call pipeline; removed public `StgInfo` and deleted thunk module; adjusted visibilities, signatures, and public exports; added numerous helpers and re-exports.
Minor codec/operator/structseq tweaks `crates/vm/src/stdlib/codecs.rs`, `crates/vm/src/stdlib/operator.rs`, `crates/vm/src/types/structseq.rs`	Replaced `is_ascii()`→`isascii()` in encode/compare paths; adjusted struct-sequence repr closure to handle `PyObject` values correctly.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Py as Python caller
    participant VM as Rust VM (_ctypes)
    participant FFI as libffi / Native
    participant Mem as Native memory

    Note right of Py: create / call ctypes objects
    Py->>VM: create type or instantiate Array/Struct/Pointer/FuncPtr
    VM->>VM: Initializer::init / process_fields / build metadata (StgInfo/type_data)
    VM->>Mem: allocate native buffer (size/align)
    Mem-->>VM: return buffer pointer
    VM-->>Py: return ctypes instance

    Note right of Py: call into native code
    Py->>VM: call PyCFuncPtr with Python args
    VM->>VM: resolve argtypes/restype, build_callargs (Python → FFI)
    VM->>FFI: invoke native function via libffi with prepared args
    FFI->>Mem: writes result/out-params to buffer
    FFI-->>VM: raw result/status
    VM->>VM: convert_raw_result (FFI → Python), extract out-parameters
    VM-->>Py: return Python result (and out-parameters)

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Areas needing extra attention:

crates/vm/src/stdlib/ctypes/function.rs — FFI conversion, call pipeline, PyCFuncPtr state and errcheck/HRESULT/COM logic.
crates/vm/src/stdlib/ctypes/{array.rs,pointer.rs,structure.rs,union.rs,simple.rs} — Initializer/SetAttr lifecycle, buffer layout/packing/alignment, pointer indexing and slice semantics, ownership/safety.
Removal/migration of StgInfo and deletion of thunk.rs — verify all consumers updated.
crates/vm/src/buffer.rs — PEP3118 nested parsing, unmatched-construct error paths.
crates/vm/src/object/core.rs — unsafe transmute/upcast helpers and invariants.

Possibly related PRs

TypeData #6403 — TypeData/StgInfo integration and TypeDataSlot APIs overlapping ctypes type-data migration.
ctypes struct/union/array #6309 — Large ctypes refactor touching pointer/POINTER creation, caching, and module wiring.
New subclass payload layout #6319 — Subclass payload/layout and upcast/downcast helpers (Py upcasting) overlap.

Suggested reviewers

arihant2math
ShaharNaveh

Poem

"🐰 I hopped through braces, bytes, and names,

Dunders snugged into their rightful frames,
I parsed the shapes PEP3118 drew,
Wrapped C memory with a careful chew,
A carrot cheer for every FFI claim!"

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	Most changes focus on ctypes implementation [ctypes/.rs], but includes unrelated refactoring in configuration files (.cspell.) and method renames across multiple modules (is_ascii→isascii, get_→____) that appear tangential to core ctypes functionality.	Separate ctypes-unrelated changes into independent PRs: spell-check dictionary updates, method renaming refactors, and buffer/protocol updates to maintain clear scope alignment with issue #2364.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'ctypes overhaul' directly and specifically describes the main change in the PR, which is a comprehensive refactoring and implementation of the ctypes module across multiple files.
Linked Issues check	✅ Passed	The PR successfully implements a comprehensive ctypes module for RustPython with Linux platform support as required by issue #2364, introducing extensive public API surface, FFI interop, buffer management, and architecture supporting future multi-platform extension.
Docstring Coverage	✅ Passed	Docstring coverage is 88.93% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}