◐ Shell
reader mode source ↗
Skip to content

bpo-45256: Avoid C calls for more Python to Python calls.#28937

Merged
markshannon merged 6 commits into
python:mainfrom
faster-cpython:flatten-all-py-calls
Oct 18, 2021
Merged

bpo-45256: Avoid C calls for more Python to Python calls.#28937
markshannon merged 6 commits into
python:mainfrom
faster-cpython:flatten-all-py-calls

Conversation

@markshannon

@markshannon markshannon commented Oct 13, 2021

Copy link
Copy Markdown
Member

Extends the approach used for CALL_FUNCTION to CALL_FUNCTION_KW, CALL_METHOD and CALL_METHOD_KW.

Also modifies initialize_locals and _PyTuple_FromArraySteal to have the same behavior w.r.t. reference counting regardless of whether they succeed or fail.

https://bugs.python.org/issue45256

@Fidget-Spinner

Copy link
Copy Markdown
Member

Did anyone discover what caused the slight slowdown in the initial PR's benchmark?

…onsume the argument references regardless of whether they succeed or fail.
@markshannon

Copy link
Copy Markdown
Member Author

Did anyone discover what caused the slight slowdown in the initial PR's benchmark?

Maybe the additional tests for consuming references?
I don't think it matters much, as specialization will handle all the fast paths.

@markshannon markshannon added skip news 🔨 test-with-buildbots Test PR w/ buildbots; report in status section labels Oct 14, 2021
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 7c0f498 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 14, 2021
@markshannon

Copy link
Copy Markdown
Member Author

@markshannon markshannon requested a review from pablogsal October 14, 2021 14:49
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 14, 2021
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 03e7ad9 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 14, 2021

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

I left some minor comments, otherwise LGTM.

Great work! I have not checked for refleaks, will do this afterwards but I also scheduled a buildbot run

@markshannon

Copy link
Copy Markdown
Member Author

I ran the buildbots earlier. All good except the Gentoo ones failing for tkinter stuff.
Feel free to re-run them, in case I broke something in the last couple of commits.

@markshannon markshannon merged commit 70945d5 into python:main Oct 18, 2021
21 hidden items Load more…
@markshannon markshannon deleted the flatten-all-py-calls branch October 18, 2021 09:55
@pablogsal

pablogsal commented Oct 27, 2021

Copy link
Copy Markdown
Member

Unfortunately, seems that this commit has broken the AMD64 FreeBSD Shared 3.x buildbot:

https://buildbot.python.org/all/#/builders/483/builds/1003/steps/5/logs/stdio

The buildbot was green until we merged this

@markshannon

Copy link
Copy Markdown
Member Author

The BSD entry in https://devguide.python.org/experts/ is empty. Not only that, but the gdb tests are decidedly uninformative when they fail. I'll see what I can come up with.

@pablogsal

Copy link
Copy Markdown
Member

I am taking a look as well, seems that the problem is that it cannot find builtin_id:

Function "builtin_id" not defined.

@pablogsal

Copy link
Copy Markdown
Member

This is one of the errors:

AssertionError:


'Breakpoint 1 (builtin_id) pending.
Breakpoint 1, builtin_id (self=<optimized out>, v=42) at Python/bltinmodule.c:1197
1197        PyObject *id = PyLong_FromVoidPtr(v);
#4 Frame 0x800235020, for file ...gdb_sample.py, line 12, in <module> ()
    foo(1, 2, 3)
Unable to find an older python frame
Locals for <module>

did not match

'^.*\nLocals for foo\na = 1\nb = 2\nc = 3\nLocals for <module>\n.*$'

@pablogsal

Copy link
Copy Markdown
Member

So seems that it was not able to find the previous python frame for some reason

@pablogsal

Copy link
Copy Markdown
Member

Something is going on, I logged into the buildbot (you need to ask koobs for access by writting to koobs@freebsd.org) and indeed many commands are broken:

140-CURRENT-amd64-564d% gdb --args ./python Lib/test/gdb_sample.py
(gdb) b builtin_id
(gdb) r
Breakpoint 1, builtin_id (self=<optimized out>, v=0x801424c00) at Python/bltinmodule.c:1197
...
(gdb) py-list
   7        baz(a, b, c)
   8
   9    def baz(*args):
  10        id(42)
  11
 >12    foo(1, 2, 3)

while on my Linux system:

❯ gdb --args ./python Lib/test/gdb_sample.py
(gdb) b builtin_id
(gdb) r
Breakpoint 1, builtin_id (self=0x7ffff7928470, v=42) at Python/bltinmodule.c:1196
1196    {
(gdb) py-list
   5
   6    def bar(a, b, c):
   7        baz(a, b, c)
   8
   9    def baz(*args):
 >10        id(42)
  11
  12    foo(1, 2, 3)
(gdb)

@pablogsal

Copy link
Copy Markdown
Member

@markshannon I know the problem. The problem is this code:

cpython/Tools/gdb/libpython.py

Lines 1804 to 1813 in d02ffd1

# gdb is unable to get the "frame" argument of PyEval_EvalFrameEx()
# because it was "optimized out". Try to get "frame" from the frame
# of the caller, _PyEval_Vector().
orig_frame = frame
caller = self._gdbframe.older()
if caller:
frame = caller.read_var('frame')
frame = PyFramePtr(frame)
if not frame.is_optimized_out():
return frame

This is not correct anymore when the frames are inlined as that frame is the top frame of all of them. The problem is that in the buildbot, the frame is optimized so it goes into this fallback.

@pablogsal

Copy link
Copy Markdown
Member

I will try to prepare a PR for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants