Issue 46555: Unicode-mangled names refer inconsistently to constants

I'm not sure if this is a bug, but it certainly surprised me. Most reserved words, when Unicode-mangled, as in "𝕕𝕖𝕗", act like ordinary identifiers (see e.g. bpo-46520). `True`, `False`, and `None` are weird in that Unicode-mangled versions of them refer to those same constants initially, but can take on their own identity as variables if assigned to:

    Python 3.9.7 (default, Sep 10 2021, 14:59:43) 
    [GCC 11.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 𝕋𝕣𝕦𝕖
    True
    >>> True = 0
      File "<stdin>", line 1
        True = 0
        ^
    SyntaxError: cannot assign to True
    >>> 𝕋𝕣𝕦𝕖 = 0
    >>> True
    True
    >>> 𝕋𝕣𝕦𝕖
    0

I think that `𝕋𝕣𝕦𝕖 = 1` should probably be forbidden. The fact that `𝕋𝕣𝕦𝕖` doesn't always mean the same thing as `True` seems to break the rule in PEP 3131 that "comparison of identifiers is based on NFKC".

hah, this is "great":

>>> 𝕋𝕣𝕦𝕖 = 1
>>> globals()
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'True': 1}

The problem is that the lexer assumes that anything that is not ASCII cannot be a keyword and lexes 𝕋𝕣𝕦𝕖 as an identifier.

True is a keyword which is compiled to expression whose value is True, 𝕋𝕣𝕦𝕖 is an identifier which refers to the builtin variable "True" which has a value True by default. You can change the value of a builtin variable, but the value of expression True is always True.

I do not see a problem here. Don't use 𝕋𝕣𝕦𝕖 if your intention is not using a variable.

> the builtin variable "True"

Is the existence of this entity, as separate from the constant `True`, documented anywhere? constants.rst doesn't seem to acknowledge it. Indeed, is its existence a feature, or is it a CPython quirk?

https://docs.python.org/3/library/constants.html#built-in-constants

Ok, I can definitely agree with Serhiy pov: "True" is a keyword that always evaluates to the object that you get when you call bool(1). There is usually no name "True" and directly assigning to it is forbidden. But there are various other ways to assign a name "True". One is eg globals("True") = 5, another one (discussed in this issue) is using identifiers that NFKC-normalize to the string "True".

Why was it decided to not raise a syntax error when the NFKC normalization of a non-ASCII token matches a keyword? I don't see a use for cases such as `𝕚𝕗 = 1` and `𝕚𝕗 + 1`. It seems the cost in terms of confusion far outweighs any potential benefit.

> Why was it decided to not raise a syntax error...

I'm not sure if such a decision was even ever made, the error happens before normalization is applied. I.e. the parser is doing two things here: (1) validating the syntax against the grammar and (2) building the AST. Normalization happens after (1), and `𝕋𝕣𝕦𝕖 = 0` is valid syntax because the grammar is NOT defined in terms of normalized identifiers, it's describing the valid (but confusing!) assignment that Carl described.

I agree that this doesn't seem like bug, but it IS my new favorite quirk of identifier normalization.