bpo-45256: Remove the usage of the C stack in Python to Python calls by pablogsal · Pull Request #28488 · python/cpython

pablogsal

Remove the usage of the cstack in Python to Python calls
Fix gdb
Move depth to frame structure for easier retrieval in debugging tools
Refactor and simplifications

https://bugs.python.org/issue45256

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit b4149b6d80bc85c73104ca0f98ee3754cf5d485c 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. A few minor issues.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nargsf and vector_nargs below are only used in asserts.
I think all the asserts and assignments to nargsf and vector_nargs could be replaced with assert(nargs == 0 || args != NULL);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Op ha

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

@markshannon Damn, unfortunately something is going on with Windows:

D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB3073: The command "setlocal
D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB3073: set PYTHONPATH=D:\a\cpython\cpython\Lib
D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB3073: "D:\a\cpython\cpython\PCbuild\win32\python.exe" "D:\a\cpython\cpython\PC\validate_ucrtbase.py" ucrtbase" exited with code -1073741819.
D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB4181: The "Exec" task returned false but did not log an error.
D:\a\cpython\cpython\PCbuild\regen.targets(107,5): error MSB3073: The command "D:\a\cpython\cpython\PCbuild\win32\python.exe Programs\freeze_test_frozenmain.py Programs/test_frozenmain.h" exited with code -1073741819. [D:\a\cpython\cpython\PCbuild\python.vcxproj]
    6 Warning(s)
    1 Error(s)

Time Elapsed 00:04:54.99

Apparently that is a File System error but I don't understand why this may be happening :(

@markshannon Oh, I found the source of the Windows failure: it was an existing bug in the PREDICT() macro for the code path without computed gotos. Fixed it in 50276ad8813bfd0bb0455d0e699e2ee301071f00

That was challenging to find indeed!

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 50276ad8813bfd0bb0455d0e699e2ee301071f00 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really cool. I just have a few comments on readability. No idea why ASAN is failing tbh.

One other concern I have is that this slows down non-python function calls very slightly (with the two additional checks). I highly doubt it's measurable on pyperformance though.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self, no need to pop frame here because it's done on eval frame exit at exit_eval_frame.

I cannot reproduce the ASAN bug. :(

There're refleaks but I can't pinpoint where :(

python.exe -m test test_grammar test_builtin -R 3:3
0:00:00 Run tests sequentially
0:00:00 [1/2] test_grammar
beginning 6 repetitions
123456
......
test_grammar leaked [12, 12, 12] references, sum=36
0:00:01 [2/2/1] test_builtin -- test_grammar failed (reference leak)
beginning 6 repetitions
123456
......
test_builtin leaked [13, 13, 13] references, sum=39
test_builtin failed (reference leak)

== Tests result: FAILURE ==

Ths commit inlines calls to Python functions in the eval loop
and steals all the arguments in the call from the caller for
performance.

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 0ddc46e25e4885562f7d1e962b9b30eea2b0283d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@markshannon what do you think of 0ddc46e for now? I dislike this approach (is correct, but is difficult to reason about if you don't have the full picture in mind). Should we do the refactor now, should we do a smaller refactor or should we just do some celanup of 0ddc46e ?

@markshannon what do you think of 0ddc46e for now? I dislike this approach (is correct, but is difficult to reason about if you don't have the full picture in mind). Should we do the refactor now, should we do a smaller refactor or should we just do some celanup of 0ddc46e ?

Just do the minimal cleanup of 0ddc46e that you are happy with for now.
And add a comment as to why it is correct.

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 1de88f6 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@markshannon I had to fix some merge conflicts and I have done the cleanup. I am building again with the buildbot fleet but this should be ready for a first version.

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit f313e8c 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

		PyObject *function = PEEK(oparg + 1);
		if (Py_TYPE(function) == &PyFunction_Type) {
		PyCodeObject code = (PyCodeObject)PyFunction_GET_CODE(function);
		PyObject *locals = code->co_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code from here onwards has waay more than 80 characters.

@Fidget-Spinner Thanks for the review!

I am not going to push anything new until the buildbots pass to not restart the long refleak builds yet again. I will fix the formatting issues in a new PR unless @markshannon wants to change something fundamental (I also plan to rename the frame->depth in another PR).

I am not going to push anything new until the buildbots pass to not restart the long refleak builds yet again.

I was just about to mention that. Good call!

		// valid arguments (i.e. the ones that fit into the frame).
		PyCodeObject co = (PyCodeObject)con->fc_code;
		const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount;
		for (Py_ssize_t i = 0; i < Py_MIN(argcount, total_args); i++) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GHA is warning:

comparison of integer expressions of different signedness: ‘Py_ssize_t’ {aka ‘long int’} and ‘long unsigned int’ [-Wsign-compare

and

'>': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]

	for (Py_ssize_t i = 0; i < Py_MIN(argcount, total_args); i++) {
	for (size_t i = 0; i < Py_MIN(argcount, (size_t)total_args); i++) {

Cleanups happening here: #28836

@markshannon After that is merged, I will create a PR to address the counter to convert it to a "entry frame" flag.