◐ Shell
reader mode source ↗
Skip to content

bpo-37751: Fix normalizestring() with hyphens and spaces converted to underscores#15092

Merged
vstinner merged 9 commits into
python:masterfrom
fusioncid:fix-issue-37751
Aug 21, 2019
Merged

bpo-37751: Fix normalizestring() with hyphens and spaces converted to underscores#15092
vstinner merged 9 commits into
python:masterfrom
fusioncid:fix-issue-37751

Conversation

@fusioncid

@fusioncid fusioncid commented Aug 3, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • Fix normalizestring() with hyphens and spaces converted to underscores
  • Reuse _Py_normalize_encoding() in normalizestring()

https://bugs.python.org/issue37751

@the-knights-who-say-ni

the-knights-who-say-ni commented Aug 3, 2019

Copy link
Copy Markdown

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

shihai1991
shihai1991 approved these changes Aug 3, 2019
View reviewed changes

@shihai1991 shihai1991 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

All the check have been passed, so LGTM.

@vstinner

vstinner commented Aug 12, 2019

Copy link
Copy Markdown
Member

Would it be possible to reuse _Py_normalize_encoding() in codecs.c normalizestring()?

@fusioncid

fusioncid commented Aug 14, 2019

Copy link
Copy Markdown
Contributor Author

Would it be possible to reuse _Py_normalize_encoding() in codecs.c normalizestring()?

I went through two functions and found that they do have similar functions. At the same time, I tried to modify the code and the test cases passed.

The code can be modified just like 'check_force_ascii‘ calls ‘_Py_normalize_encoding‘ . And I think it's better to call _Py_normalize_encoding as an external function to other modules.

There is also a process problem. I am not quite sure whether to submit another issue to discuss this modification, or can I directly modify it in this issue?

@fusioncid

fusioncid commented Aug 14, 2019

Copy link
Copy Markdown
Contributor Author

Would it be possible to reuse _Py_normalize_encoding() in codecs.c normalizestring()?

I will do more test for reusing _Py_normalize_encoding() in codecs.c normalizestring(). Thank you for your helpful suggestions。

@vstinner

vstinner commented Aug 14, 2019

Copy link
Copy Markdown
Member

You can create a new PR, this one can be closed when the new one is merged.

@fusioncid

fusioncid commented Aug 15, 2019

Copy link
Copy Markdown
Contributor Author

You can create a new PR, this one can be closed when the new one is merged.

Let me try it.

@fusioncid fusioncid changed the title bpo-37751: Fix normalizestring() with hyphens and spaces converted to… Aug 17, 2019
@bedevere-bot

bedevere-bot commented Aug 19, 2019

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@fusioncid @vstinner
Update Python/codecs.c …
e51724b 
I think use a regular exception is better. Thanks.

Co-Authored-By: Victor Stinner <vstinner@redhat.com>
@vstinner

vstinner commented Aug 19, 2019

Copy link
Copy Markdown
Member

Can you please try to add a NEWS entry using the blurb tool? (install it using: python3 -m pip install --user blurb)

@fusioncid

fusioncid commented Aug 19, 2019

Copy link
Copy Markdown
Contributor Author

Can you please try to add a NEWS entry using the blurb tool? (install it using: python3 -m pip install --user blurb)

Thank you for your careful guidance. I need to take a moment to familiarize myself with the blurb tool.

@fusioncid

fusioncid commented Aug 20, 2019

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again.

11 hidden items Load more…
@bedevere-bot

bedevere-bot commented Aug 20, 2019

Copy link
Copy Markdown

Thanks for making the requested changes!

@vstinner: please review the changes made to this pull request.

vstinner
vstinner approved these changes Aug 21, 2019
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM. Thanks for the update.

@vstinner vstinner merged commit 20f59fe into python:master Aug 21, 2019
@fusioncid fusioncid deleted the fix-issue-37751 branch August 21, 2019 14:24
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@fusioncid @lisroach
bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092) …
74447e6 
Fix codecs.lookup() to normalize the encoding name the same way
than encodings.normalize_encoding(), except that codecs.lookup()
also converts the name to lower case.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@fusioncid @DinoV
bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092) …
49d58a2 
Fix codecs.lookup() to normalize the encoding name the same way
than encodings.normalize_encoding(), except that codecs.lookup()
also converts the name to lower case.
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
@fusioncid @websurfer5
bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092) …
22de191 
Fix codecs.lookup() to normalize the encoding name the same way
than encodings.normalize_encoding(), except that codecs.lookup()
also converts the name to lower case.
@yan12125 yan12125 mentioned this pull request Dec 2, 2020
Merged
eight04 added a commit to eight04/pyUAO that referenced this pull request Feb 22, 2021
@eight04
Add: support Python 3.9 (#1) …
68a8f54 
* Fix running with Python 3.9

Since Python 3.9 [1], codecs names are normalized in a different way.

[1] python/cpython#15092

* Add: github action, bump dependencies

Co-authored-by: eight04 <eight04@gmail.com>
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Jul 28, 2025
@serhiy-storchaka
pythongh-88886: Remove excessive encoding name normalization …
02ae0e1 
The codecs lookup function now performs only minimal normalization of
the encoding name before passing it to the serach functions:
all ASCII letters are converted to lower case, spaces are replaced
with hyphens.

Excessive normalization broke third-party codecs providers, like
python-iconv.

Revert "bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092)"

This reverts commit 20f59fe.
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Jul 28, 2025
@serhiy-storchaka
pythongh-88886: Remove excessive encoding name normalization …
2df6169 
The codecs lookup function now performs only minimal normalization of
the encoding name before passing it to the search functions:
all ASCII letters are converted to lower case, spaces are replaced
with hyphens.

Excessive normalization broke third-party codecs providers, like
python-iconv.

Revert "bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092)"

This reverts commit 20f59fe.
@serhiy-storchaka serhiy-storchaka mentioned this pull request Jul 28, 2025
Merged
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Jul 28, 2025
@serhiy-storchaka
pythongh-88886: Remove excessive encoding name normalization …
ae1cae2 
The codecs lookup function now performs only minimal normalization of
the encoding name before passing it to the search functions:
all ASCII letters are converted to lower case, spaces are replaced
with hyphens.

Excessive normalization broke third-party codecs providers, like
python-iconv.

Revert "bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092)"

This reverts commit 20f59fe.
serhiy-storchaka added a commit that referenced this pull request Sep 9, 2025
@serhiy-storchaka
gh-88886: Remove excessive encoding name normalization (GH-137167) …
af58a6f 
The codecs lookup function now performs only minimal normalization of
the encoding name before passing it to the search functions:
all ASCII letters are converted to lower case, spaces are replaced
with hyphens.

Excessive normalization broke third-party codecs providers, like
python-iconv.

Revert "bpo-37751: Fix codecs.lookup() normalization (GH-15092)"

This reverts commit 20f59fe.
lkollar pushed a commit to lkollar/cpython that referenced this pull request Sep 9, 2025
@serhiy-storchaka @lkollar
pythongh-88886: Remove excessive encoding name normalization (pythonG… …
0a3e79b 
…H-137167)

The codecs lookup function now performs only minimal normalization of
the encoding name before passing it to the search functions:
all ASCII letters are converted to lower case, spaces are replaced
with hyphens.

Excessive normalization broke third-party codecs providers, like
python-iconv.

Revert "bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092)"

This reverts commit 20f59fe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes

@shihai1991 shihai1991 shihai1991 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@fusioncid @the-knights-who-say-ni @vstinner @bedevere-bot @shihai1991