bpo-40334: PEP 617: New PEG parser for CPython by pablogsal · Pull Request #19503 · python/cpython

- The pegen parser generator and its tests are in Tools/peg_generator. This is mostly copied from the https://github.com/gvanrossum/pegen repo, with some adaptations. This is still targeting 3.8.
- The new grammar (adapted to Python 3.9) is in Grammar/python.gram.
- There's a new builtin module peg_parser, configured in Setup.
- There's a new Make target regen-pegen which runs the parser generator with output directed to Modules/peg_parser/parse.c.

There's some code duplication between Tools/peg_generator and Modules/peg_parser, due to the 3.8/3.9 split and some tweaks needed to link with the core. Eventually this probably should go away.

* Use trailer meta for a function that calls the start rule
* Expose four C API functions that return either a mod_ty or a PyCodeObject *
* Expose two Python-level functions that return an AST object
* Extension module and C API does not get generated, but is hand-written
* Rewrite of run_parser* functions

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

* The caller is responsible for transforming the mod_ty a code or an AST object
* The C API functions don't expect the filename and the arena (where possible) objects as an
   argument, but construct them themselves

* Rewrite exported function:
    * It now initializes the keywords field so that the keywords variable can be static
    * It also accepts an argument, in order to be able to run the parser with different start rules
* Use the newly created function in fstring_compile_expr

…enerator itself (#23)

* Remove C code from Tools/peg_generator/peg_parser and rely on Parser/pegen
* Move all tests to Lib/test/test_peg_generator/*
* Disable pegen tests on Github Actions

* Remove start rule and add file, interactive, eval and fstring rules
* Accept a mode parameter in C API functions

…active mode (#38)

* Refactor raise_syntax_error to correctly handle interactive mode
* Correctly handle Ctrl-D in interactive mode

Additionally, fix the script to calculate the nesting level using the generated extension.

* Fix audit tests
* Try to fallback to get_error_line if we cannot get the error line using PyErr_ProgramTextObject
* Add -E flag to run_tests.py only if PYTHONPARSER is not set

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>