GH-130328: pasting in new REPL is slow on Windows by chris-eibl · Pull Request #132884 · python/cpython

chris-eibl

The reason why pasting is so slow on Windows (especially in the legacy console case where virtual terminal mode - and thus bracketed paste - is disabled):

	while True:
	# We use the same timeout as in readline.c: 100ms
	self.run_hooks()
	self.console.wait(100)
	event = self.console.get_event(block=False)

and

	def wait(self, timeout: float \| None) -> bool:
	"""Wait for an event."""
	# Poor man's Windows select loop
	start_time = time.time()
	while True:
	if msvcrt.kbhit(): # type: ignore[attr-defined]
	return True
	if timeout and time.time() - start_time > timeout / 1000:
	return False
	time.sleep(0.01)

It is interesting, that msvcrt.kbhit() returns True in case of pasting, but if that weren't the case, we'd hit the 100ms timeout and would be even way slower. However, msvcrt.kbhit() is very slow in case of the legacy console, and it is called at least twice as often, because we get a key up and a key down event - but only a key down event in the virtual terminal case. If the pasted input contains upper case letters, we additionally get a "shift key pressed" event - so in the worst case, msvcrt.kbhit() gets called 3 times more often (and each call is slower).

Output of python.bat -m cProfile -m _pyrepl when pasting the "test value" given in the OP for a legacy console:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     50/1    0.010    0.000   17.202   17.202 {built-in method builtins.exec}
        1    0.000    0.000   17.202   17.202 <frozen runpy>:201(run_module)
        1    0.000    0.000   17.199   17.199 <frozen runpy>:65(_run_code)
        1    0.000    0.000   17.199   17.199 __main__.py:1(<module>)
        1    0.000    0.000   17.112   17.112 main.py:24(interactive_console)
        1    0.000    0.000   17.112   17.112 simple_interact.py:99(run_multiline_interactive_console)
        5    0.000    0.000   17.111    3.422 readline.py:375(multiline_input)
        5    0.004    0.001   17.111    3.422 reader.py:741(readline)
     3090    0.036    0.000   17.103    0.006 reader.py:694(handle1)
     6210    0.013    0.000   13.714    0.002 windows_console.py:523(wait)
     6236   13.433    0.002   13.433    0.002 {built-in method msvcrt.kbhit}

Here the relevant line in case of virtual terminal:

  3270    0.554    0.000    0.554    0.000 {built-in method msvcrt.kbhit}

The fix is to use WaitForSingleObject

    def wait(self, timeout: float | None) -> bool:
        """Wait for an event."""
        ret = WaitForSingleObject(InHandle, int(timeout))
        if ret == WAIT_FAILED:
            raise WinError(ctypes.get_last_error())

which speeds up especially the legacy console (times in seconds):

	legacy terminal	virtual terminal
before	15.6	0.89
after	1.8	0.26

Note, see also #132440 (comment):

legacy terminal: manually start cmd.exe. Timings gotten by pasting

import time
t1 = time.time()
<"test value" given in the OP>
print(time.time() - t1)

virtual terminal: power shell via Windows terminal. Timings gotton by temporarily patching commands.py:

import time
class enable_bracketed_paste(Command):
    def do(self) -> None:
        self.reader.bp_begin = time.perf_counter()
        self.reader.paste_mode = True
        self.reader.in_bracketed_paste = True

class disable_bracketed_paste(Command):
    def do(self) -> None:
        print("bracketed_paste took %5.2f s" % (time.perf_counter() - self.reader.bp_begin, ))
        self.reader.paste_mode = False
        self.reader.in_bracketed_paste = False
        self.reader.dirty = True

Issue: Pasting in new REPL is slow #130328