◐ Shell
clean mode source ↗

Message 293793 - Python tracker

> I prefer to add tokenize tokens directly in Lib/token.py, and then get COMMENT, NL and ENCODING using tok_name.index().

That would make more sense from a breaking change perspective, but we would step on the toes of anyone adding `COMMENT`, `NL`, or `ENCODING` to `token.h` because `token.py` is generated from that.

It would also make much more sense to have them as fields on `token` if they are in `tok_name` in `token`.