bpo-29812: Improve testing of token and tokenize#681
Conversation
Sorry, something went wrong.
|
@ammaraskar, thanks for your PR! By analyzing the history of the files in this pull request, we identified @serhiy-storchaka, @tiran and @1st1 to be potential reviewers. |
Sorry, something went wrong.
ammaraskar
force-pushed
the
master
branch
2 times, most recently
from
5c9e66a to
dbce4b5
Compare
March 16, 2017 02:49
|
Weirdly enough this fails on AppVeyor (which is Windows, I believe). I'm not sure if this is something wrong with the test or an actual problem on windows, I'll look into it tomorrow when I have access to a windows computer. |
Sorry, something went wrong.
ammaraskar
force-pushed
the
master
branch
3 times, most recently
from
963035d to
9b357ae
Compare
April 11, 2017 20:26
|
Oh also, I tried to look into what was calling the AppVeyor failure when comparing the files. From what I can tell its because os.stat is returning an incorrect value for st_size on the test environment or something https://ci.appveyor.com/project/ammaraskar/cpython/build/1.0.9#L3949 Doing a file content comparison causes it to pass. Nevermind, I don't think this is an incorrect return since the offset of the size is exactly the number of lines, I think this has to do with line endings. |
Sorry, something went wrong.
Adds a token.py generation test, and a cross-check between the token and tokenize modules. The generation test is akin to the current automated file generator tests in test_keyword and test_symbol. The newly added check in tokenize ensures that all the literal tokens in token such as ':', '...', '@=' have a corresponding entry within the EXACT_TOKEN_TYPES dict in tokenize.
Currently this code does not work on windows
when the skeleton file has unix line endings
because fp.write('\n') will have the LF
changed to CRLF due to python's newline
interpolation.
|
Hey @berkerpeksag, could you take a look at this, I updated it fixing the concerns in your review. @bitdancer could I also get a review from you seeing as you made the issue on the tracker for this? |
Sorry, something went wrong.
berkerpeksag
left a comment
berkerpeksag
left a comment
There was a problem hiding this comment.
Sorry, something went wrong.
|
Thanks for your review @berkerpeksag. Closing this as per discussion on bpo. |
Sorry, something went wrong.
Adds a token.py generation test, and a cross-check between the token and tokenize modules.
The generation test is akin to the current automated file generator tests in test_keyword and test_symbol. The newly added check in tokenize ensures that all the literal tokens in token such as ':', '...', '@=' have a corresponding entry within the EXACT_TOKEN_TYPES dict in tokenize.