◐ Shell
reader mode source ↗
Skip to content

gh-94823: Improve coverage in tokenizer.c:valid_utf8#94856

Merged
miss-islington merged 4 commits into
python:mainfrom
mdboom:tokenizer-valid-utf8-coverage
Aug 16, 2022
Merged

gh-94823: Improve coverage in tokenizer.c:valid_utf8#94856
miss-islington merged 4 commits into
python:mainfrom
mdboom:tokenizer-valid-utf8-coverage

Conversation

@mdboom

@mdboom mdboom commented Jul 14, 2022 •
edited by miss-islington
Loading

Copy link
Copy Markdown
Contributor

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in unicode_decode_utf8. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.

Automerge-Triggered-By: GH:ericsnowcurrently

@bedevere-bot bedevere-bot added tests Tests in the Lib/test dir awaiting review labels Jul 14, 2022
@brandtbucher brandtbucher added needs backport to 3.11 only security fixes labels Jul 15, 2022
@mdboom
pythongh-94823: Improve coverage in tokenizer.c:valid_utf8 …
e52f328 
When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in `unicode_decode_utf8`. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.
@mdboom mdboom force-pushed the tokenizer-valid-utf8-coverage branch from 6db7d84 to e52f328 Compare July 15, 2022 18:11
ericsnowcurrently
ericsnowcurrently requested changes Aug 12, 2022
View reviewed changes

@ericsnowcurrently ericsnowcurrently left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Thanks for working on this! It mostly looks good. I noticed a couple of small typos.

@bedevere-bot

bedevere-bot commented Aug 12, 2022

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@mdboom

mdboom commented Aug 16, 2022

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

@bedevere-bot

bedevere-bot commented Aug 16, 2022

Copy link
Copy Markdown

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

ericsnowcurrently
ericsnowcurrently approved these changes Aug 16, 2022
View reviewed changes

@ericsnowcurrently ericsnowcurrently left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM

@miss-islington miss-islington merged commit f215d7c into python:main Aug 16, 2022
@miss-islington

miss-islington commented Aug 16, 2022

Copy link
Copy Markdown
Contributor

Thanks @mdboom for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 16, 2022
@mdboom @miss-islington
pythongh-94823: Improve coverage in tokenizer.c:valid_utf8 (pythonGH-… …
0425b23 
…94856)

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in `unicode_decode_utf8`. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.
(cherry picked from commit f215d7c)

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Aug 16, 2022
@bedevere-bot

bedevere-bot commented Aug 16, 2022

Copy link
Copy Markdown

GH-96029 is a backport of this pull request to the 3.11 branch.

@ericsnowcurrently

ericsnowcurrently commented Aug 16, 2022

Copy link
Copy Markdown
Member

Thanks for the test!

pablogsal pushed a commit that referenced this pull request Aug 16, 2022
@miss-islington @mdboom
[3.11] gh-94823: Improve coverage in tokenizer.c:valid_utf8 (GH-94856) ( …
2bb363c 
#96029)

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
This was referenced Aug 22, 2022
Closed
Closed
Merged
@kumaraditya303 kumaraditya303 mentioned this pull request Sep 17, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes

Assignees

No one assigned

Labels

skip news tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@mdboom @bedevere-bot @miss-islington @ericsnowcurrently @brandtbucher @kumaraditya303