Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Thanks for the contribution! Would you mind adding a regression test for this in https://github.com/python/cpython/blob/main/Lib/test/test_tokenize.py

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

I added a test.

(discussed with PSRT) We're not considering this a security issue, compile() is not safe against untrusted input (nor are the other paths that are actually executing code rather than just compiling it). It can read a couple bytes past the end of the input bytes at most. at worst a crash (ASAN in particular which I believe is how oss-fuzz found it), you'd likely get a SyntaxError as intended otherwise.

We do like to have issues filed to track things - read up on https://devguide.python.org/getting-started/pull-request-lifecycle/#pullrequest
As I believe you're intending to fix up a bunch of fuzzer found items, consider making a tracking issue for similar themes of those "parser crashes" or similar for this one for example - we don't need one issue per item when they aren't actually security related.

while you're working on already public oss-fuzz findings, as they are public it is fine to go ahead and make public PRs.

But if you come across any that seem actually security relevant, please reach out via email to the to PSRT, or since you're producing fix PRs maybe just use https://github.com/python/cpython/security to drop those in GHSA's for us to look over (many will probably be things we just open directly to the public as a non-concern "more of a bug" like this one)

All commit authors signed the Contributor License Agreement.

@aisk thank you, is it me or you who needs to sign the CLA?

@aisk thank you, is it me or you who needs to sign the CLA?

it was me.

Thanks @AdamKorcz for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

Sorry, @AdamKorcz and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 3fc945df22a169e039c3f21b44c0d08390a00c0c 3.13

GH-145287 is a backport of this pull request to the 3.14 branch.

@AdamKorcz can you do the 3.13 backport following #144807 (comment) ?

GH-145441 is a backport of this pull request to the 3.13 branch.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 NoGIL 3.13 (tier-2) has failed when building commit 5577bd1.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1408/builds/1843) and take a look at the build logs.
4. Check if the failure is related to this commit (5577bd1) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1408/builds/1843

Failed tests:

• test_httpservers
• test.test_concurrent_futures.test_thread_pool

Failed subtests:

• test_free_reference - test.test_concurrent_futures.test_thread_pool.ThreadPoolExecutorTest.test_free_reference
• test_large_content_length_truncated - test.test_httpservers.CGIHTTPServerTestCase.test_large_content_length_truncated

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_concurrent_futures/executor.py", line 147, in test_free_reference
    for _ in support.sleeping_retry(support.SHORT_TIMEOUT):
             ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/support/__init__.py", line 2596, in sleeping_retry
    for _ in busy_retry(timeout, err_msg, error=error):
             ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/support/__init__.py", line 2566, in busy_retry
    raise AssertionError(msg)
AssertionError: timeout (45.9 seconds)


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_concurrent_futures/executor.py", line 147, in test_free_reference
    for _ in support.sleeping_retry(support.SHORT_TIMEOUT):
             ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/support/__init__.py", line 2596, in sleeping_retry
    for _ in busy_retry(timeout, err_msg, error=error):
             ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/support/__init__.py", line 2566, in busy_retry
    raise AssertionError(msg)
AssertionError: timeout (45.8 seconds)


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_httpservers.py", line 1028, in test_large_content_length_truncated
    res = self.request('/cgi-bin/file1.py', 'POST', b'x', headers)
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_httpservers.py", line 89, in request
    return self.connection.getresponse()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/http/client.py", line 1450, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/http/client.py", line 336, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/http/client.py", line 297, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.13.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
ConnectionResetError: [Errno 54] Connection reset by peer