◐ Shell
reader mode source ↗
Skip to content

gh-96268: Fix loading invalid UTF-8#96270

Merged
gvanrossum merged 11 commits into
python:mainfrom
mdboom:fix-valid-utf8
Sep 7, 2022
Merged

gh-96268: Fix loading invalid UTF-8#96270
gvanrossum merged 11 commits into
python:mainfrom
mdboom:fix-valid-utf8

Conversation

@mdboom

@mdboom mdboom commented Aug 25, 2022 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Contributor

This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

This also fixes the related test so it will always detect the expected failure
and error message.

mdboom added 2 commits August 25, 2022 10:13
@mdboom
pythongh-96268: Fix loading invalid UTF-8 …
e4aaa14 
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

This also fixes the related test so it will always detect the expected failure
and error message.
@mdboom
Add blurb
407eef7
@bedevere-bot bedevere-bot added the label Aug 25, 2022
@mdboom mdboom added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section topic-unicode needs backport to 3.11 only security fixes and removed awaiting review labels Aug 25, 2022
@bedevere-bot

bedevere-bot commented Aug 25, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @mdboom for commit 407eef7 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@mdboom mdboom requested a review from ezio-melotti August 25, 2022 16:25
@gvanrossum gvanrossum self-requested a review August 29, 2022 16:52
gvanrossum
gvanrossum reviewed Aug 30, 2022
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Got me nerd-sniped. :-)

@mdboom mdboom requested a review from gvanrossum August 31, 2022 14:55
@mdboom

mdboom commented Aug 31, 2022

Copy link
Copy Markdown
Contributor Author

@pablogsal: I leave it to you to decide whether this is backported to 3.11. If we don't backport, I'll file a separate PR for 3.11 to make the tests pass on buildbots with pydebug and saving coredump files (where they are currently failing).

gvanrossum
gvanrossum approved these changes Aug 31, 2022
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

I'll let @pablogsal decide about the 3.11 and 3.10 backports. (It would be less risky to backport just the lineno fix perhaps?)

@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 31, 2022
@bedevere-bot

bedevere-bot commented Aug 31, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit f8e9e6e 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 31, 2022
@mdboom mdboom mentioned this pull request Sep 6, 2022
Closed
gvanrossum
gvanrossum approved these changes Sep 7, 2022
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Thanks. I think it's time to merge this.

@gvanrossum gvanrossum merged commit 8bc356a into python:main Sep 7, 2022
@miss-islington

miss-islington commented Sep 7, 2022

Copy link
Copy Markdown
Contributor

Thanks @mdboom for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

@bedevere-bot

bedevere-bot commented Sep 7, 2022

Copy link
Copy Markdown

GH-96668 is a backport of this pull request to the 3.11 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 7, 2022
@mdboom @miss-islington
pythongh-96268: Fix loading invalid UTF-8 (pythonGH-96270) …
4bca34e 
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

It also fixes an off-by-one error introduced in 3.10 for the line number when the tokenizer reports bad UTF8.
(cherry picked from commit 8bc356a)

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
miss-islington added a commit that referenced this pull request Sep 7, 2022
@miss-islington @mdboom
gh-96268: Fix loading invalid UTF-8 (GH-96270) …
ffafa9b 
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

It also fixes an off-by-one error introduced in 3.10 for the line number when the tokenizer reports bad UTF8.
(cherry picked from commit 8bc356a)

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gvanrossum gvanrossum gvanrossum approved these changes

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti

Assignees

No one assigned

Labels

topic-unicode

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@mdboom @bedevere-bot @miss-islington @gvanrossum @pablogsal @ezio-melotti