bpo-34515: lib2to3: support non-ASCII identifiers by holymonson · Pull Request #8950 · python/cpython

the-knights-who-say-ni

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lib/tokenize.py appears to parse numbers before identifiers to avoid having a look-behind assertion here. Can we take that approach here, too?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjaminp copied codes in Lib/tokenize.py.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Lib/tokenize.py, I see:

Thanks @holymonson for the PR, and @benjaminp for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

Sep 15, 2018

…-8950)

(cherry picked from commit 10a428b)

Co-authored-by: Monson Shao <holymonson@gmail.com>

@benjaminp What we actually want is to merge the two pure Python tokenizers. This pull request makes this harder.

That does seem like a better solution. Do you want me to revert this?

I'm thinking. My change will only affect Python 3.8 so the backport PR (GH-9333) does make life of YAPF users better in the interim. I'll revert on master only when I rebase my tokenizer merge pull request.

IOW, let's leave it for now.

miss-islington added a commit that referenced this pull request

Sep 15, 2018

(cherry picked from commit 10a428b)

Co-authored-by: Monson Shao <holymonson@gmail.com>