bpo-25643: Refactor the C tokenizer by pablogsal · Pull Request #25050 · python/cpython

@pablogsal, @serhiy-storchaka: I added some PEP 7 comments, some comments about deprecated memory functions, and some decref / free control questions. Hope you don't mind.

Thanks a lot for your comments and for the review. I have applied most of the PEP7 and made some changes to error returning paths so they are a bit more clear.

Regarding PEP7, normally we only adapt it when we are reworking things, but given that this change was quite big I had opted to not make it worse by changing the surrounding style, but I think it doesn't hurt that much.