◐ Shell
reader mode source ↗
Skip to content

bpo-40334: Refactor peg_generator to receive a Tokens file when building c code#19745

Merged
pablogsal merged 4 commits into
python:masterfrom
pablogsal:bpo-40334-use-tokens
Apr 28, 2020
Merged

bpo-40334: Refactor peg_generator to receive a Tokens file when building c code#19745
pablogsal merged 4 commits into
python:masterfrom
pablogsal:bpo-40334-use-tokens

Conversation

@pablogsal

@pablogsal pablogsal commented Apr 28, 2020 •
edited
Loading

Copy link
Copy Markdown
Member

https://bugs.python.org/issue40334

This PR does the following:

  • Fix a bunch of (very minor) mypy stuff that was missing.
  • Separate the C parser and the Python parser in pegen main (because both receive different arguments). Thread down all these changes to the generator build module.
  • Add a new option to the C parser command line to receive the Tokens file.
  • Thread down the Tokens file and add code to parse it and calculate the required token information.
  • Use the new tokens in the c_generator (and simplify some code that was hardcoding some token names).
  • Update the build files (Makefile and the Windows one) to use the new option.
  • Run black over the source.

@pablogsal pablogsal requested a review from a team as a code owner April 28, 2020 00:15
@pablogsal pablogsal force-pushed the bpo-40334-use-tokens branch from 6c13937 to 45ca184 Compare April 28, 2020 00:16
@pablogsal

pablogsal commented Apr 28, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

This is how the command line looks now with the sub-parsers:

Main CL

~/github/python/master/Tools/peg_generator [bpo-40334](https://bugs.python.org/issue40334)-use-tokens
❯ python -m pegen
usage: pegen [-h] [-q] [-v] [--skip-actions] {c,python} ...

Experimental PEG-like parser generator

positional arguments:
  {c,python}      target language for the generated code
    c             Generate C code for inclusion into CPython
    python        Generate Python code

optional arguments:
  -h, --help      show this help message and exit
  -q, --quiet     Don't print the parsed grammar
  -v, --verbose   Print timing stats; repeat for more debug output

C subparser

~/github/python/master/Tools/peg_generator [bpo-40334](https://bugs.python.org/issue40334)-use-tokens
❯ python -m pegen c -h
usage: pegen c [-h] [--compile-extension] [-o OUT] [--optimized] [--skip-actions] grammar_filename tokens_filename

positional arguments:
  grammar_filename      Grammar description
  tokens_filename       Tokens description

optional arguments:
  -h, --help            show this help message and exit
  -o OUT, --output OUT  Where to write the generated parser
  --compile-extension   Compile generated C code into an extension module
  --optimized           Compile the extension in optimized mode
  --skip-actions        Suppress code emission for rule actions

Python subparser

~/github/python/master/Tools/peg_generator [bpo-40334](https://bugs.python.org/issue40334)-use-tokens
❯ python -m pegen python -h
usage: pegen python [-h] [-o OUT] grammar_filename

positional arguments:
  grammar_filename      Grammar description

optional arguments:
  -h, --help            show this help message and exit
  -o OUT, --output OUT  Where to write the generated parser
  --skip-actions  Suppress code emission for rule actions

@pablogsal pablogsal force-pushed the bpo-40334-use-tokens branch 2 times, most recently from 1aab4d1 to d7df6b6 Compare April 28, 2020 00:29
@pablogsal pablogsal force-pushed the bpo-40334-use-tokens branch from d7df6b6 to a371b79 Compare April 28, 2020 00:30
pablogsal and others added 3 commits April 28, 2020 12:34
@pablogsal @lysnikolaou
Apply suggestions from code review …
417d503 
Co-Authored-By: Lysandros Nikolaou <lisandrosnik@gmail.com>
@pablogsal @lysnikolaou
Update Tools/peg_generator/pegen/c_generator.py …
c268ef1 
Co-Authored-By: Lysandros Nikolaou <lisandrosnik@gmail.com>
@pablogsal
Address feedback
9b8dfd8
@pablogsal pablogsal requested a review from lysnikolaou April 28, 2020 11:51
lysnikolaou
lysnikolaou approved these changes Apr 28, 2020
View reviewed changes

@lysnikolaou lysnikolaou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM! Thanks!

@pablogsal pablogsal merged commit 5b9f498 into python:master Apr 28, 2020
@bedevere-bot

This comment has been minimized.

Sign in to view
@pablogsal

pablogsal commented Apr 28, 2020

Copy link
Copy Markdown
Member Author

Hi! The buildbot AMD64 Fedora Stable 3.x has failed when building commit 5b9f498.

Unrelated failure:

1 test failed:
test_concurrent_futures

@pablogsal pablogsal mentioned this pull request Apr 28, 2020
Merged
@pablogsal pablogsal deleted the bpo-40334-use-tokens branch May 1, 2020 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lysnikolaou lysnikolaou lysnikolaou approved these changes

@gvanrossum gvanrossum Awaiting requested review from gvanrossum

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pablogsal @bedevere-bot @lysnikolaou @the-knights-who-say-ni