bpo-40334: Refactor peg_generator to receive a Tokens file when building c code by pablogsal · Pull Request #19745 · python/cpython

pablogsal

https://bugs.python.org/issue40334

This PR does the following:

Fix a bunch of (very minor) mypy stuff that was missing.
Separate the C parser and the Python parser in pegen main (because both receive different arguments). Thread down all these changes to the generator build module.
Add a new option to the C parser command line to receive the Tokens file.
Thread down the Tokens file and add code to parse it and calculate the required token information.
Use the new tokens in the c_generator (and simplify some code that was hardcoding some token names).
Update the build files (Makefile and the Windows one) to use the new option.
Run black over the source.

This is how the command line looks now with the sub-parsers:

Main CL

~/github/python/master/Tools/peg_generator [bpo-40334](https://bugs.python.org/issue40334)-use-tokens
❯ python -m pegen
usage: pegen [-h] [-q] [-v] [--skip-actions] {c,python} ...

Experimental PEG-like parser generator

positional arguments:
  {c,python}      target language for the generated code
    c             Generate C code for inclusion into CPython
    python        Generate Python code

optional arguments:
  -h, --help      show this help message and exit
  -q, --quiet     Don't print the parsed grammar
  -v, --verbose   Print timing stats; repeat for more debug output

C subparser

~/github/python/master/Tools/peg_generator [bpo-40334](https://bugs.python.org/issue40334)-use-tokens
❯ python -m pegen c -h
usage: pegen c [-h] [--compile-extension] [-o OUT] [--optimized] [--skip-actions] grammar_filename tokens_filename

positional arguments:
  grammar_filename      Grammar description
  tokens_filename       Tokens description

optional arguments:
  -h, --help            show this help message and exit
  -o OUT, --output OUT  Where to write the generated parser
  --compile-extension   Compile generated C code into an extension module
  --optimized           Compile the extension in optimized mode
  --skip-actions        Suppress code emission for rule actions

Python subparser

~/github/python/master/Tools/peg_generator [bpo-40334](https://bugs.python.org/issue40334)-use-tokens
❯ python -m pegen python -h
usage: pegen python [-h] [-o OUT] grammar_filename

positional arguments:
  grammar_filename      Grammar description

optional arguments:
  -h, --help            show this help message and exit
  -o OUT, --output OUT  Where to write the generated parser
  --skip-actions  Suppress code emission for rule actions