Implement Windows named mmap via CreateFileMappingW#7207
Conversation
📝 WalkthroughWalkthroughThis PR adds Windows named memory-mapping support: it exposes the Win32 System Memory feature in Cargo.toml and implements a Windows-only NamedMmap type plus associated code paths to create, map, access, flush, and drop named file mappings. Changes
Sequence Diagram(s)sequenceDiagram
participant Python as "Python user"
participant PyMmap as "PyMmap"
participant Validator as "Tagname validator"
participant Win32 as "Win32 APIs"
participant Kernel as "Kernel memory"
Python->>PyMmap: py_new(..., tagname="name")
PyMmap->>Validator: validate/convert tagname
Validator-->>PyMmap: validated tagname
PyMmap->>Win32: CreateFileMappingW(..., name)
Win32->>Kernel: allocate/identify named mapping
Kernel-->>Win32: map_handle
Win32-->>PyMmap: map_handle
PyMmap->>Win32: MapViewOfFile(map_handle, access)
Win32->>Kernel: map view into process
Kernel-->>Win32: view_ptr
Win32-->>PyMmap: view_ptr
PyMmap->>PyMmap: store NamedMmap(map_handle, view_ptr, len)
PyMmap-->>Python: return MmapObj::Named
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Sorry, something went wrong.
|
Code has been automatically formatted The code in this PR has been formatted using:
git pull origin win-mmap |
Sorry, something went wrong.
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [ ] test: cpython/Lib/test/test_mmap.py (TODO: 2) dependencies: dependent tests: (no tests depend on mmap) Legend:
|
Sorry, something went wrong.
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/stdlib/src/mmap.rs (1)
1190-1219: ⚠️ Potential issue | 🔴 Critical
resizesilently converts aNamedmmap into an anonymous one.For
Namedmmaps,self.handleisINVALID_HANDLE_VALUE, sois_anonymousistrue. The anonymous resize path creates a newMmapMutviamap_anon()and replaces theNamedvariant withMmapObj::Write. This silently destroys the named shared memory mapping — any other process sharing the tagname will no longer see updates, and the mapping loses its named identity.At minimum,
Namedmmaps should be detected and an error raised (similar to how the Unix path returns an error for missingmremap). Better yet, the named mapping should be re-created withCreateFileMappingW+MapViewOfFileat the new size.Proposed minimal fix: reject resize for Named mmaps
// Get the lock on mmap let mut mmap_guard = self.mmap.lock(); + // Named mmaps cannot be resized via the anonymous path + if let Some(MmapObj::Named(_)) = mmap_guard.as_ref() { + return Err(vm.new_system_error( + "mmap: resizing named memory mappings is not supported", + )); + } + if is_anonymous {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@crates/stdlib/src/mmap.rs` around lines 1190 - 1219, The resize implementation silently converts Named mappings into anonymous ones by checking is_anonymous via self.handle and then unconditionally calling map_anon() and assigning MmapObj::Write; instead detect Named mappings in resize (use self.handle, MmapObj variants and mmap.lock()/mmap_guard) and return an error when attempting to resize a Named mmap (mirror the Unix behavior for missing mremap) OR implement re-creation via Windows APIs (CreateFileMappingW + MapViewOfFile) to preserve the named mapping; update the code path that currently calls MmapOptions::map_anon() so it either raises a clear error for Named maps or uses the CreateFileMappingW/MapViewOfFile flow to create a new named mapping and copy data before replacing mmap_guard and updating self.size.
🧹 Nitpick comments (1)
crates/stdlib/src/mmap.rs (1)
849-861:as_bytes_mutreturns a mutable slice forNamedregardless of access mode.Unlike
MmapObj::Readwhich panics, theNamedarm always returns a mutable slice. While current callers (try_writable, buffer protocol) check access mode before reaching this point, adding a debug assertion would strengthen the invariant:#[cfg(windows)] - MmapObj::Named(named) => unsafe { - core::slice::from_raw_parts_mut(named.view_ptr, named.len) - }, + MmapObj::Named(named) => { + debug_assert!( + !matches!(self.access, AccessMode::Read), + "mmap can't modify a readonly memory map." + ); + unsafe { core::slice::from_raw_parts_mut(named.view_ptr, named.len) } + }Note:
selfis not directly accessible inas_bytes_mutsince it operates on theMmapObjmatch. If restructuring isn't feasible, adding a comment documenting the caller-enforced invariant would suffice.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@crates/stdlib/src/mmap.rs` around lines 849 - 861, The as_bytes_mut method currently returns a mutable slice for MmapObj::Named regardless of access mode; update as_bytes_mut (the match arm for MmapObj::Named) to either add a debug assertion that the named mapping is writable (assert the Named object's access/flags indicate writable access) or, if that field isn't accessible here, add a clear comment above the method and the MmapObj::Named arm documenting the caller-enforced invariant (try_writable and buffer protocol callers must have verified writability) so future readers/maintainers know why a mutable slice is returned despite no runtime check; reference as_bytes_mut, MmapObj::Named, MmapObj::Read, try_writable and the buffer protocol in the comment or assertion.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@crates/stdlib/src/mmap.rs`:
- Around line 1016-1019: The comment is wrong: MmapObj::Named can be file-backed
and thus may need flushing; update the flush handling for the Windows Named
variant in the flush implementation to either detect whether the Named mapping
was created from a real file handle (file-backed) and call FlushViewOfFile only
for those, or simply call FlushViewOfFile unconditionally for MmapObj::Named.
Locate the Windows branch handling MmapObj::Named in the flush function and
replace the no-op with a call to the appropriate Windows flush routine
(FlushViewOfFile) or conditional logic based on how the Named mapping was
constructed.
- Around line 671-680: The current logic closes duplicated_handle and leaves
self.handle as INVALID_HANDLE_VALUE for file-backed named mappings (code around
duplicated_handle, fh and CreateFileMappingW), which changes behavior of size()
and resize(); either preserve the duplicated handle in self.handle or document
the intentional divergence. Fix by not closing duplicated_handle when fh is Some
and tagname is used: keep duplicated_handle open and assign it to self.handle
after successful CreateFileMappingW (ensure you only close duplicated_handle on
error or when the mmap is dropped), update the cleanup paths to avoid
double-closing, and add a short comment near the
duplicated_handle/fh/CreateFileMappingW block explaining why the duplicated OS
file handle is retained in self.handle so size() and resize() continue to
operate on the file-backed mapping.
- Around line 663-728: The code computes total_size = (offset as u64) +
(map_size as u64) without overflow checks in the tagged-path; change this to use
checked_add like the non-tagged path (use offset as u64 .checked_add(map_size as
u64)) and on None return the same kind of error (e.g., return
Err(io::Error::new(...).to_pyexception(vm))) before calling CreateFileMappingW
so CreateFileMappingW and subsequent logic (CreateFileMappingW, MapViewOfFile,
NamedMmap construction) only run with a validated, non-overflowing size.
---
Outside diff comments:
In `@crates/stdlib/src/mmap.rs`:
- Around line 1190-1219: The resize implementation silently converts Named
mappings into anonymous ones by checking is_anonymous via self.handle and then
unconditionally calling map_anon() and assigning MmapObj::Write; instead detect
Named mappings in resize (use self.handle, MmapObj variants and
mmap.lock()/mmap_guard) and return an error when attempting to resize a Named
mmap (mirror the Unix behavior for missing mremap) OR implement re-creation via
Windows APIs (CreateFileMappingW + MapViewOfFile) to preserve the named mapping;
update the code path that currently calls MmapOptions::map_anon() so it either
raises a clear error for Named maps or uses the CreateFileMappingW/MapViewOfFile
flow to create a new named mapping and copy data before replacing mmap_guard and
updating self.size.
---
Nitpick comments:
In `@crates/stdlib/src/mmap.rs`:
- Around line 849-861: The as_bytes_mut method currently returns a mutable slice
for MmapObj::Named regardless of access mode; update as_bytes_mut (the match arm
for MmapObj::Named) to either add a debug assertion that the named mapping is
writable (assert the Named object's access/flags indicate writable access) or,
if that field isn't accessible here, add a clear comment above the method and
the MmapObj::Named arm documenting the caller-enforced invariant (try_writable
and buffer protocol callers must have verified writability) so future
readers/maintainers know why a mutable slice is returned despite no runtime
check; reference as_bytes_mut, MmapObj::Named, MmapObj::Read, try_writable and
the buffer protocol in the comment or assertion.
ℹ️ Review info
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
Lib/test/test_mmap.pyis excluded by!Lib/**
📒 Files selected for processing (2)
crates/stdlib/Cargo.tomlcrates/stdlib/src/mmap.rs
Sorry, something went wrong.
Use Win32 CreateFileMappingW + MapViewOfFile when tagname is provided, enabling inter-process shared memory for multiprocessing.heap.Arena. Anonymous mmaps without tagname still use memmap2. Remove expectedFailure from test_tagname in test_mmap.py.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
crates/stdlib/src/mmap.rs (1)
671-680: ⚠️ Potential issue | 🟠 MajorFile-backed named mappings lose the OS file handle —
size()won't return the file size.When a tagname is provided alongside a real file handle, the duplicated handle is closed and
self.handleis set toINVALID_HANDLE_VALUE. This causessize()(line 1331) to returnself.__len__()(the mapping length) instead of querying the actual file size viaGetFileSize. CPython'smmap.size()returns the file size for file-backed mappings.Consider storing
duplicated_handleinself.handlefor file-backed named mmaps instead of closing it, or add a comment explaining the intentional divergence from CPython behavior.Also applies to: 720-729
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@crates/stdlib/src/mmap.rs` around lines 671 - 680, The code closes duplicated_handle and leaves self.handle as INVALID_HANDLE_VALUE for file-backed named mappings, causing size() to fall back to the mapping length instead of using GetFileSize; change the logic in the block that currently returns fh/INVALID_HANDLE_VALUE (the code handling fh and duplicated_handle) so that for file-backed named mmaps you keep the duplicated_handle (assign duplicated_handle to self.handle) instead of closing it, and only close the original duplicated_handle when it is truly unused; ensure the new ownership is consistent with the destructor/cleanup so no handle is leaked and size() can call GetFileSize on self.handle.
🧹 Nitpick comments (1)
crates/stdlib/src/mmap.rs (1)
851-863:as_bytes_mutdoesn't guard against Read-mode Named mappings.
MmapObj::Readpanics here (line 854), but aNamedmmap created withAccessMode::Read(FILE_MAP_READ) will silently return a mutable slice. Writing to that memory would cause an OS-level access violation. While callers currently check access mode upstream, adding a consistent guard here would be safer.🛡️ Suggested defensive guard
#[cfg(windows)] MmapObj::Named(named) => unsafe { + debug_assert!( + !matches!( + *m.as_ref().unwrap(), + MmapObj::Named(_) if false // access check is on PyMmap, not MmapObj + ) + ); core::slice::from_raw_parts_mut(named.view_ptr, named.len) },Actually, since
as_bytes_mutdoesn't have access toself.access, a simpler approach: document that callers must check access mode, mirroring theMmapObj::Readpanic's implicit contract.Alternatively, pass
accessinto the function or move the check to the match:fn as_bytes_mut(&self) -> BorrowedValueMut<'_, [u8]> { + assert!( + self.access != AccessMode::Read, + "mmap can't modify a readonly memory map." + ); PyMutexGuard::map(self.mmap.lock(), |m| { match m.as_mut().expect("mmap closed or invalid") { MmapObj::Read(_) => panic!("mmap can't modify a readonly memory map."), MmapObj::Write(mmap) => &mut mmap[..],🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@crates/stdlib/src/mmap.rs` around lines 851 - 863, as_bytes_mut currently returns a mutable slice for MmapObj::Named even if that named mapping was opened read-only, which can lead to an access violation; update as_bytes_mut (or its caller) to check the mapping's access mode and refuse/mirror the MmapObj::Read behavior for read-only Named maps: either accept and use an access parameter in as_bytes_mut and panic when access == AccessMode::Read for MmapObj::Named, or add a guard in the match arm for MmapObj::Named that inspects the named mapping's access flag and panics if it's read-only, ensuring Named maps opened with AccessMode::Read are not exposed as mutable slices.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@crates/stdlib/src/mmap.rs`:
- Around line 227-243: The Drop impl for NamedMmap uses .is_null() on map_handle
which is an isize (windows-sys HANDLE) and doesn't have that method; change the
checks that call self.map_handle.is_null() to integer comparisons (e.g.
self.map_handle != 0 or self.map_handle == 0) while leaving pointer checks like
self.view_ptr.is_null() as-is; update the Drop impl (and any other places using
map_handle.is_null(), e.g. the other occurrence noted) to use map_handle != 0
before calling CloseHandle.
---
Duplicate comments:
In `@crates/stdlib/src/mmap.rs`:
- Around line 671-680: The code closes duplicated_handle and leaves self.handle
as INVALID_HANDLE_VALUE for file-backed named mappings, causing size() to fall
back to the mapping length instead of using GetFileSize; change the logic in the
block that currently returns fh/INVALID_HANDLE_VALUE (the code handling fh and
duplicated_handle) so that for file-backed named mmaps you keep the
duplicated_handle (assign duplicated_handle to self.handle) instead of closing
it, and only close the original duplicated_handle when it is truly unused;
ensure the new ownership is consistent with the destructor/cleanup so no handle
is leaked and size() can call GetFileSize on self.handle.
---
Nitpick comments:
In `@crates/stdlib/src/mmap.rs`:
- Around line 851-863: as_bytes_mut currently returns a mutable slice for
MmapObj::Named even if that named mapping was opened read-only, which can lead
to an access violation; update as_bytes_mut (or its caller) to check the
mapping's access mode and refuse/mirror the MmapObj::Read behavior for read-only
Named maps: either accept and use an access parameter in as_bytes_mut and panic
when access == AccessMode::Read for MmapObj::Named, or add a guard in the match
arm for MmapObj::Named that inspects the named mapping's access flag and panics
if it's read-only, ensuring Named maps opened with AccessMode::Read are not
exposed as mutable slices.
ℹ️ Review info
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
Lib/test/test_mmap.pyis excluded by!Lib/**
📒 Files selected for processing (2)
crates/stdlib/Cargo.tomlcrates/stdlib/src/mmap.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/stdlib/Cargo.toml
Sorry, something went wrong.
490a17f
into
RustPython:main
Feb 23, 2026
Use Win32 CreateFileMappingW + MapViewOfFile when tagname is provided, enabling inter-process shared memory for multiprocessing.heap.Arena. Anonymous mmaps without tagname still use memmap2. Remove expectedFailure from test_tagname in test_mmap.py.
Use Win32 CreateFileMappingW + MapViewOfFile when tagname is provided, enabling inter-process shared memory for multiprocessing.heap.Arena. Anonymous mmaps without tagname still use memmap2.
Remove expectedFailure from test_tagname in test_mmap.py.
Summary by CodeRabbit