bpo-29812: Improve testing of token and tokenize by ammaraskar · Pull Request #681 · python/cpython

the-knights-who-say-ni

Adds a token.py generation test, and a cross-check between the
token and tokenize modules.

The generation test is akin to the current automated file generator tests
in test_keyword and test_symbol. The newly added check in tokenize ensures
that all the literal tokens in token such as ':', '...', '@=' have a
corresponding entry within the EXACT_TOKEN_TYPES dict in tokenize.

Currently this code does not work on windows
when the skeleton file has unix line endings
because fp.write('\n') will have the LF
changed to CRLF due to python's newline
interpolation.