gh-91048: Refactor and optimize remote debugging module#134652
Conversation
Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase.
ambv
left a comment
There was a problem hiding this comment.
It's not easy to look at the _remote_debugging_module.c diff given the number of changes, but I went through the module in a checkout and it looks much cleaner now.
I like the caching, I like free-threading handling, and the Unwinder object. I found a race in your test, I'll fix that for you alongside some trivial pep8 nits.
Sorry, something went wrong.
42b25ad
into
python:main
May 25, 2025
This comment was marked as outdated.
This comment was marked as outdated.
…#134652) Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase. Also, make the module compatible with the free threading build as an extra :) Co-authored-by: Łukasz Langa <lukasz@langa.pl> (cherry picked from commit 42b25ad)
…) (#134673) gh-91048: Refactor and optimize remote debugging module (#134652) Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase. Also, make the module compatible with the free threading build as an extra :) Co-authored-by: Łukasz Langa <lukasz@langa.pl> (cherry picked from commit 42b25ad)
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot iOS ARM64 Simulator 3.14 (tier-3) has failed when building commit 1822f33. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/#/builders/1722/builds/161 Summary of the results of the build (if available): Click to see traceback logsremote: Enumerating objects: 60, done.
remote: Counting objects: 1% (1/60)
remote: Counting objects: 3% (2/60)
remote: Counting objects: 5% (3/60)
remote: Counting objects: 6% (4/60)
remote: Counting objects: 8% (5/60)
remote: Counting objects: 10% (6/60)
remote: Counting objects: 11% (7/60)
remote: Counting objects: 13% (8/60)
remote: Counting objects: 15% (9/60)
remote: Counting objects: 16% (10/60)
remote: Counting objects: 18% (11/60)
remote: Counting objects: 20% (12/60)
remote: Counting objects: 21% (13/60)
remote: Counting objects: 23% (14/60)
remote: Counting objects: 25% (15/60)
remote: Counting objects: 26% (16/60)
remote: Counting objects: 28% (17/60)
remote: Counting objects: 30% (18/60)
remote: Counting objects: 31% (19/60)
remote: Counting objects: 33% (20/60)
remote: Counting objects: 35% (21/60)
remote: Counting objects: 36% (22/60)
remote: Counting objects: 38% (23/60)
remote: Counting objects: 40% (24/60)
remote: Counting objects: 41% (25/60)
remote: Counting objects: 43% (26/60)
remote: Counting objects: 45% (27/60)
remote: Counting objects: 46% (28/60)
remote: Counting objects: 48% (29/60)
remote: Counting objects: 50% (30/60)
remote: Counting objects: 51% (31/60)
remote: Counting objects: 53% (32/60)
remote: Counting objects: 55% (33/60)
remote: Counting objects: 56% (34/60)
remote: Counting objects: 58% (35/60)
remote: Counting objects: 60% (36/60)
remote: Counting objects: 61% (37/60)
remote: Counting objects: 63% (38/60)
remote: Counting objects: 65% (39/60)
remote: Counting objects: 66% (40/60)
remote: Counting objects: 68% (41/60)
remote: Counting objects: 70% (42/60)
remote: Counting objects: 71% (43/60)
remote: Counting objects: 73% (44/60)
remote: Counting objects: 75% (45/60)
remote: Counting objects: 76% (46/60)
remote: Counting objects: 78% (47/60)
remote: Counting objects: 80% (48/60)
remote: Counting objects: 81% (49/60)
remote: Counting objects: 83% (50/60)
remote: Counting objects: 85% (51/60)
remote: Counting objects: 86% (52/60)
remote: Counting objects: 88% (53/60)
remote: Counting objects: 90% (54/60)
remote: Counting objects: 91% (55/60)
remote: Counting objects: 93% (56/60)
remote: Counting objects: 95% (57/60)
remote: Counting objects: 96% (58/60)
remote: Counting objects: 98% (59/60)
remote: Counting objects: 100% (60/60)
remote: Counting objects: 100% (60/60), done.
remote: Compressing objects: 7% (1/14)
remote: Compressing objects: 14% (2/14)
remote: Compressing objects: 21% (3/14)
remote: Compressing objects: 28% (4/14)
remote: Compressing objects: 35% (5/14)
remote: Compressing objects: 42% (6/14)
remote: Compressing objects: 50% (7/14)
remote: Compressing objects: 57% (8/14)
remote: Compressing objects: 64% (9/14)
remote: Compressing objects: 71% (10/14)
remote: Compressing objects: 78% (11/14)
remote: Compressing objects: 85% (12/14)
remote: Compressing objects: 92% (13/14)
remote: Compressing objects: 100% (14/14)
remote: Compressing objects: 100% (14/14), done.
remote: Total 31 (delta 27), reused 19 (delta 17), pack-reused 0 (from 0)
From https://github.com/python/cpython
* branch 3.14 -> FETCH_HEAD
Note: switching to '1822f33b1af989ebb9e7a5b58bd40721fca9a218'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 1822f33b1af [3.14] gh-91048: Refactor and optimize remote debugging module (#134652) (#134673)
Switched to and reset branch '3.14'
configure: WARNING: no system libmpdecimal found; falling back to bundled libmpdecimal (deprecated and scheduled for removal in Python 3.15)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
configure: WARNING: no system libmpdecimal found; falling back to bundled libmpdecimal (deprecated and scheduled for removal in Python 3.15)
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
In file included from ../../Modules/_remote_debugging_module.c:31:
../../Modules/../Python/remote_debug.h:139:20: error: call to undeclared function 'pid_to_task'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
139 | handle->task = pid_to_task(handle->pid);
| ^
1 error generated.
make: *** [Modules/_remote_debugging_module.o] Error 1 |
Sorry, something went wrong.
|
iOS build failure seems to be due to a discrepancy between the |
Sorry, something went wrong.
…#134652) Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase. Also, make the module compatible with the free threading build as an extra :) Co-authored-by: Łukasz Langa <lukasz@langa.pl>
…#134652) Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase. Also, make the module compatible with the free threading build as an extra :) Co-authored-by: Łukasz Langa <lukasz@langa.pl>
…#134652) Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase. Also, make the module compatible with the free threading build as an extra :) Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API.
Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance.
Add code object caching mechanism with a new code_object_generation field that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions.
Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process.
Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase.