◐ Shell
clean mode source ↗

Issue 40334: PEP 617: new PEG-based parser

Created on 2020-04-19 20:16 by gvanrossum, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (110) msg366803 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-19 20:16
Note: PEP 617 is currently under review by the Steering Council, but if they approve we'd like to get it into alpha 6, and reviews are welcome (even though we're still finagling some corner cases of the implementation).
msg367009 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-22 15:16
Once PR 19503 will be merged, I would be interested to see if setjmp()/longjmp() can be avoided. I'm scared by these functions: they require that all functions in between setjmp() and longjmp() call stack have no state and don't need any cleanup at exit.
msg367050 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-22 21:07
VIctor, you were very right about longjmp. See https://github.com/we-like-parsers/cpython/pull/119
msg367057 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-22 22:29
New changeset c5fc15685202cda73f7c3f5c6f299b0945f58508 by Pablo Galindo in branch 'master':
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
https://github.com/python/cpython/commit/c5fc15685202cda73f7c3f5c6f299b0945f58508
msg367059 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-22 23:13
New changeset 458004bf7914f96b20bb76bc3584718cf83f652e by Pablo Galindo in branch 'master':
bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)
https://github.com/python/cpython/commit/458004bf7914f96b20bb76bc3584718cf83f652e
msg367060 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-22 23:19
> bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)

Pablo told me in private that he plans to add a buildbot worker to test the old parser. So this change is fine.
msg367061 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-22 23:21
Looks like the build changes do not work with a build directory outside of the source directory:

mkdir build && cd build && ../configure && make

gcc -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include    -DPy_BUILD_CORE -o Parser/pegen/pegen.o ../Parser/pegen/pegen.c
error: unable to open output file 'Parser/pegen/pegen.o': 'No such file or directory'
1 error generated.
make: *** [Parser/pegen/pegen.o] Error 1

Lots of build processes depend on this, like for putting out a release.
msg367063 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-22 23:42
> Looks like the build changes do not work with a build directory outside of the source directory:

Opened https://github.com/python/cpython/pull/19667. Ned, could you review?
msg367064 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 00:38
New changeset a25f3c4c8f7d4878918ce1d3d67db40ae255ccc6 by Pablo Galindo in branch 'master':
bpo-40334: Fix builds outside the source directory and regenerate autoconf files (GH-19667)
https://github.com/python/cpython/commit/a25f3c4c8f7d4878918ce1d3d67db40ae255ccc6
msg367066 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 01:03
New changeset 1def7754b7a41fe57efafaf5eff24cfa15353444 by Victor Stinner in branch 'master':
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
https://github.com/python/cpython/commit/1def7754b7a41fe57efafaf5eff24cfa15353444
msg367067 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 01:09
> bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)

This change removes sys.flags.use_peg which was only used in tests. If someone sees a good reason, we may add it back. In the meanwhile, test.support.use_old_parser() can be used.

I didn't like the idea of having a new sys.flags.use_peg flag appears in 3.9 but disappears in 3.10.

FYI sys.flags still supports the tuple API! Example:

>>> sys.flags[9]
0

I have no idea what is this 10th member of sys.flags :-)
msg367068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 01:15
test_peg_generator emits compiler warnings:

https://buildbot.python.org/all/#/builders/612/builds/283

0:08:01 load avg: 1.40 [423/423/1] test_peg_generator passed (7 min 4 sec)
/tmp/tmp0vbqe8gp/parse.c: In function ‘start_rule’:
/tmp/tmp0vbqe8gp/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
   32 |             _PyPegen_lookahead(1, _PyPegen_name_token, p)
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   struct _expr * (*)(Parser *)
In file included from /tmp/tmp0vbqe8gp/parse.c:2:
/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (*)(Parser *)’ but argument is of type ‘struct _expr * (*)(Parser *)’
   90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      |                             ~~~~~~~^~~~~~~~~~~~~~~
/tmp/tmp54o68k3n/parse.c: In function ‘start_rule’:
/tmp/tmp54o68k3n/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
   32 |             _PyPegen_lookahead(0, _PyPegen_name_token, p)
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   struct _expr * (*)(Parser *)
In file included from /tmp/tmp54o68k3n/parse.c:2:
/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (*)(Parser *)’ but argument is of type ‘struct _expr * (*)(Parser *)’
   90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      |                             ~~~~~~~^~~~~~~~~~~~~~~
msg367069 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-23 01:18
> test_peg_generator emits compiler warnings

These are expected. Is this a problem?
msg367071 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-23 01:55
> These are expected. Is this a problem?

People and bots running tests normally expect to not see any output from each test in the default case unless there are errors.  If these are supposed to be emitted, we should find a way to hide them (and check the results, if necessary) unless a verbose option is selected.  Personally, I wouldn't consider that a showstopper for a6 but as something to be fixed by b1.
msg367073 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 02:43
New changeset ee40e4b8563e6e1bc2bfb267da5ffc9a2293318d by Pablo Galindo in branch 'master':
bpo-40334: Don't downcast from Py_ssize_t to int (GH-19671)
https://github.com/python/cpython/commit/ee40e4b8563e6e1bc2bfb267da5ffc9a2293318d
msg367075 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 02:51
> test_peg_generator emits compiler warnings:

This is fixed by https://github.com/python/cpython/pull/19672
msg367076 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 02:52
Those warnings are legitimate: we cannot cast function pointers that have different return types from one to the other. This only happens in the test, but could happen in the grammar if we use lookahead with NAME (&NAME).
msg367084 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-23 06:52
Issue40370 documents test_peg_generator breakage on an AIX buildbot.
msg367097 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 11:42
New changeset 1df5a9e88c8df1495a2e689d71b34bf0c7d008ea by Pablo Galindo in branch 'master':
bpo-40334: Fix build errors and warnings in test_peg_generator (GH-19672)
https://github.com/python/cpython/commit/1df5a9e88c8df1495a2e689d71b34bf0c7d008ea
msg367105 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 12:22
New changeset 8d1cbfffea7a5dbf7a3c60b066a2c2120ef08cd0 by Lysandros Nikolaou in branch 'master':
bpo-40334: Suppress all output in test_peg_generator (GH-19675)
https://github.com/python/cpython/commit/8d1cbfffea7a5dbf7a3c60b066a2c2120ef08cd0
msg367112 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 14:05
GCC warning on aarch64 Fedora Rawhide LTO + PGO 3.x buildbot:
https://buildbot.python.org/all/#/builders/612/builds/286

In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5709:25,
    inlined from ‘assemble’ at Python/compile.c:6048:18:
Python/compile.c:5663:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5663 |         *lnotab++ = k;
      |         ~~~~~~~~~~^~~
msg367113 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 14:10
> GCC warning on aarch64 Fedora Rawhide LTO + PGO 3.x buildbot:

Is this related to this issue? We didn't change that line
msg367114 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-04-23 14:17
> Is this related to this issue? We didn't change that line

I can provide you access to the buildbot if you'd like to debug the issue.
msg367115 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 14:28
> Is this related to this issue? We didn't change that line

I don't know. It's just that I spotted this compiler warnining while reviewing recent buildbot failures.
msg367121 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 15:17
> I don't know. It's just that I spotted this compiler warnining while reviewing recent buildbot failures.

Given that is a compiler warning and we didn't change those lines, I would say is not related to this PR :)
msg367126 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 15:36
New changeset ebebb6429c224c713e1c63a0b05d4840f52c7415 by Lysandros Nikolaou in branch 'master':
bpo-40334: Improve various PEG-Parser related stuff (GH-19669)
https://github.com/python/cpython/commit/ebebb6429c224c713e1c63a0b05d4840f52c7415
msg367135 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 18:06
AMD64 Windows10 3.x:
https://buildbot.python.org/all/#builders/129/builds/825

Warning -- files was modified by test_peg_generator
  Before: []
  After:  ['parse_d.cp39-win_amd64.pdb', 'vc140.pdb']
msg367136 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-23 18:12
> Warning -- files was modified by test_peg_generator
>   Before: []
>   After:  ['parse_d.cp39-win_amd64.pdb', 'vc140.pdb']

Currently working on this.
msg367137 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-23 18:13
$ make -s regen-all
./Include/opcode.h.new regenerated from ./Lib/opcode.py
Jump table written into ./Python/opcode_targets.h.new
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 14, in <module>
    from typing import Final
ImportError: cannot import name 'Final'
Makefile:837: recipe for target 'regen-pegen' failed
make: *** [regen-pegen] Error 1
msg367139 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 18:38
> ImportError: cannot import name 'Final'

Currently, you need Python3.8+ to do 'make regen-pegen'. We can try to modify the module to not use newer features like "Final" so you can use older Pythons.
msg367140 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-23 18:42
It does not work with python3.8 either.

$ make -s regen-all PYTHON_FOR_REGEN=python3.8
./Include/opcode.h.new regenerated from ./Lib/opcode.py
Jump table written into ./Python/opcode_targets.h.new
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 16, in <module>
    from pegen.build import build_parser_and_generator
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/build.py", line 13, in <module>
    from distutils.tests.support import fixup_build_ext
ModuleNotFoundError: No module named 'distutils.tests'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 16, in <module>
    from pegen.build import build_parser_and_generator
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/build.py", line 13, in <module>
    from distutils.tests.support import fixup_build_ext
ModuleNotFoundError: No module named 'distutils.tests'
Makefile:837: recipe for target 'regen-pegen' failed
make: *** [regen-pegen] Error 1
msg367141 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 18:48
> oduleNotFoundError: No module named 'distutils.tests'

Oh, are you using a python distribution that strips distutils? Is this Debian or Ubuntu?

I suppose we need to account for that. I will create a PR for trying to address both. That's for pointing that out, Serhiy!
msg367142 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 18:49
"That's for pointing that out" -> "Thanks for pointing that out"
msg367155 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-23 22:43
New changeset bc28805570ae3e8835a5d502ae9ab15c52449f77 by Guido van Rossum in branch 'master':
bpo-40334: Use old compiler when compile mode is func_type (GH-19692)
https://github.com/python/cpython/commit/bc28805570ae3e8835a5d502ae9ab15c52449f77
msg367158 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 22:57
I don't see any mention of the PEP 617 in What's New in Python 3.9:
https://docs.python.org/dev/whatsnew/3.9.html
msg367160 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 23:53
New changeset 50f28dea32c45e1a49b3bd07c874b4fa837a5e88 by Pablo Galindo in branch 'master':
bpo-40334: Allow to run make regen-pegen without distutils (GH-19684)
https://github.com/python/cpython/commit/50f28dea32c45e1a49b3bd07c874b4fa837a5e88
msg367170 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-24 02:33
test_peg_generator leaks references.

Example with s390x RHEL8 Refleaks 3.x:
https://buildbot.python.org/all/#builders/536/builds/67

test_peg_generator leaked [24246, 24242, 24246] references, sum=72734
test_peg_generator leaked [10928, 10926, 10928] memory blocks, sum=32782


The following test is enough to trigger a leak:

$ ./python -m test -R 3:3 test_peg_generator -m test.test_peg_generator.test_c_parser.TestCParser.test_error_in_rules

0:00:00 load avg: 0.94 Run tests sequentially
0:00:00 load avg: 0.94 [1/1] test_peg_generator
beginning 6 repetitions
123456
......
test_peg_generator leaked [55, 55, 55] references, sum=165
test_peg_generator leaked [19, 19, 19] memory blocks, sum=57
test_peg_generator failed

== Tests result: FAILURE ==

1 test failed:
    test_peg_generator

Total duration: 10.8 sec
Tests result: FAILURE
msg367171 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-24 03:56
Yeah that's why we have a PR out to delete or skip this test. Pablo and
Lysandros will deal with it tomorrow.

On Thu, Apr 23, 2020 at 19:33 STINNER Victor <report@bugs.python.org> wrote:

>
> STINNER Victor <vstinner@python.org> added the comment:
>
> test_peg_generator leaks references.
>
> Example with s390x RHEL8 Refleaks 3.x:
> https://buildbot.python.org/all/#builders/536/builds/67
>
> test_peg_generator leaked [24246, 24242, 24246] references, sum=72734
> test_peg_generator leaked [10928, 10926, 10928] memory blocks, sum=32782
>
>
> The following test is enough to trigger a leak:
>
> $ ./python -m test -R 3:3 test_peg_generator -m
> test.test_peg_generator.test_c_parser.TestCParser.test_error_in_rules
>
> 0:00:00 load avg: 0.94 Run tests sequentially
> 0:00:00 load avg: 0.94 [1/1] test_peg_generator
> beginning 6 repetitions
> 123456
> ......
> test_peg_generator leaked [55, 55, 55] references, sum=165
> test_peg_generator leaked [19, 19, 19] memory blocks, sum=57
> test_peg_generator failed
>
> == Tests result: FAILURE ==
>
> 1 test failed:
>     test_peg_generator
>
> Total duration: 10.8 sec
> Tests result: FAILURE
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40334>
> _______________________________________
>
-- 
--Guido (mobile)
msg367190 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-24 13:51
New changeset 24ffe705c30e36c82940d75fd1454256634d0b3c by Lysandros Nikolaou in branch 'master':
bpo-40334: Rewrite test_c_parser to avoid memory leaks (GH-19694)
https://github.com/python/cpython/commit/24ffe705c30e36c82940d75fd1454256634d0b3c
msg367191 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-24 13:53
> test_peg_generator leaks references.

This is now fixed.
msg367196 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-24 14:56
Lysandros:
> This is now fixed.

Thanks, that was quick ;-)
msg367237 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2020-04-24 23:09
:waves: -- seeing a lot of failures in pyflakes' testsuite around SyntaxErrors https://github.com/PyCQA/pyflakes/pull/532

(you can reproduce with `path/to/python -m unittest discover pyflakes` without needing any dependencies installed)
msg367243 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-25 00:20
New changeset 0e80b561d442769631d66f1cc8813ac30f97378e by Guido van Rossum in branch 'master':
bpo-40334: Add What's New sections for PEP 617 and PEP 585 (GH-19704)
https://github.com/python/cpython/commit/0e80b561d442769631d66f1cc8813ac30f97378e
msg367246 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-25 00:45
@Anthony: Presumably some of these will be unavoidable. I see you have PYPY-specific checks in your code already.

You could also help by sending us the test programs and the different output produced by the old and the new parser. You could either create a Gist and link it here, or open bug reports here: https://github.com/we-like-parsers/cpython/issues . I opened one already for you: https://github.com/we-like-parsers/cpython/issues/121
msg367439 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-27 17:02
New changeset 2b74c835a7280840a853e3a9aaeb83758b13a458 by Pablo Galindo in branch 'master':
bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)
https://github.com/python/cpython/commit/2b74c835a7280840a853e3a9aaeb83758b13a458
msg367440 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-27 17:36
New changeset b94dbd7ac34dc0c79512656eb17f6f07e09fca7a by Pablo Galindo in branch 'master':
bpo-40334: Support PyPARSE_DONT_IMPLY_DEDENT in the new parser (GH-19736)
https://github.com/python/cpython/commit/b94dbd7ac34dc0c79512656eb17f6f07e09fca7a
msg367448 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-27 19:21
The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

It should either parse the correct Grammar/Tokens file, or read the content of corresponding files Lib/token.py and Lib/tokenize.py and evaluate them with eval(). See for example Tools/scripts/generate_token.py.
msg367467 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-27 22:15
> The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

Very good point, Serhiy! Thanks for catching that

So there are two parts of the parser generator where we use these modules:

- For the grammar parser and the python-based generator (that generates the grammar parser) we are good because we just need the modules to parse the grammars and generate the metaparser so there is no need for those modules to be updated. Running the grammar parser (the one generated with the meta-grammar) is also fine.

- For the C generator we need the current set of "exact_token_types" and in this case we need them to be synchronized with the Tokens file. I think we can do as pgen and take the path to that file as part of the command line interface and parse it.

I will make a PR for this soon.
msg367470 - (view) Author: paul rubin (phr) Date: 2020-04-27 23:52
I just saw this.  Interesting.  Sometimes I use ast.literal_eval to read big, deeply nested data objects.  I can probably convert to JSON if necessary but it's another thing to watch out for.  I might try to benchmark some of these.
msg367471 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-27 23:55
@phr: examples welcome! The bigger and nastier, the better. This *should* all be speedy but it wouldn't hurt to check.
msg367473 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 00:23
New changeset d55133f49fe678fbf047a647aa8bb8b520410e8d by Lysandros Nikolaou in branch 'master':
bpo-40334: Catch E_EOF error, when the tokenizer returns ERRORTOKEN (GH-19743)
https://github.com/python/cpython/commit/d55133f49fe678fbf047a647aa8bb8b520410e8d
msg367474 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 00:24
New changeset 3d53d8756f0403eec6a4e12f183103d651bed6c5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Don't skip test_parser:test_trigget_memory_error (GH-19744)
https://github.com/python/cpython/commit/3d53d8756f0403eec6a4e12f183103d651bed6c5
msg367521 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 12:12
New changeset 5b9f4988c94f47fa35e84f154a7b5aa17bc04722 by Pablo Galindo in branch 'master':
bpo-40334: Refactor peg_generator to receive a Tokens file when building c code (GH-19745)
https://github.com/python/cpython/commit/5b9f4988c94f47fa35e84f154a7b5aa17bc04722
msg367564 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 21:27
On x86 Gentoo Non-Debug with X 3.x buildbot, test_peg_generator timed out twice after 15 minutes (first time when tests are run in parallel, then when the test is run sequentially).

Maybe the buildbot worker is just slow. On a previous build (749), test_peg_generator timed out after 15 min (parallel run), but then passed in 12 minutes when re-run in verbose mode.

The buildbot worker should get a larger timeout. I suggest to increase the timeout, or the test should be optimized to take less time.

Pablo: Would you mind to increase ware-gentoo-x86 timeout? Ex: 30 min instead of 15 min.

--

The failed build 750:
https://buildbot.python.org/all/#/builders/188/builds/750

0:58:45 load avg: 4.59 [423/423/2] test_peg_generator crashed (Exit code 1)
(... many compiler warnings ...)
Timeout (0:15:00)!
Thread 0xb7bba700 (most recent call first):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1873 in _try_wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1915 in _wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1185 in wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/spawn.py", line 75 in spawn
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 910 in spawn
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/unixccompiler.py", line 117 in _compile
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 574 in compile
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 529 in build_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 474 in _build_extensions_serial
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 449 in build_extensions
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 340 in run
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/build.py", line 89 in compile_c_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/testutil.py", line 101 in generate_parser_c_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 76 in build_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 79 in run_test
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 240 in test_return_stmt_noexpr_action
  (...)
msg367576 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 23:43
> Pablo: Would you mind to increase ware-gentoo-x86 timeout? Ex: 30 min instead of 15 min.


Done: https://github.com/python/buildmaster-config/pull/190
msg367577 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 23:45
It seems like building Python on Windows 64-bit emits a few warnings:

D:\a\1\s\Parser\pegen\pegen.c(610,91): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
D:\a\1\s\Parser\pegen\pegen.c(611,52): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
D:\a\1\s\Parser\pegen\pegen.c(612,103): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
msg367580 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 23:58
The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431):

    clear = "NORMAL"
    print(dict(state=clear, bg="#d00" if clear else"#fca"))

Well, the code looks like a typo error... but it worked previously. Not sure if we should keep the SyntaxError or not. Fixing the code is trivial: see PR 19777 attached to bpo-40431; replace >else"#fca"< with >else "#fca"<.
msg367582 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 00:00
> The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431):


This is not due to the new parser but due to https://bugs.python.org/issue40246. Please, report it there as well (you can reopen that issue for further discussion).
msg367598 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 00:43
New changeset 37af21b667a9f41437b5b8e451497d7725016df5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix shifting of nested f-strings in the new parser (GH-19771)
https://github.com/python/cpython/commit/37af21b667a9f41437b5b8e451497d7725016df5
msg367602 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 01:04
New changeset 2208134918ee673451e4fc525bbeab71142d794a by Pablo Galindo in branch 'master':
bpo-40334: Explicitly cast to int in pegen.c to fix a compiler warning (GH-19779)
https://github.com/python/cpython/commit/2208134918ee673451e4fc525bbeab71142d794a
msg367610 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 01:42
New changeset 6d6508765514c7c10719478a0430f5e47c9a96ac by Lysandros Nikolaou in branch 'master':
bpo-40334: Disallow invalid single statements in the new parser (GH-19774)
https://github.com/python/cpython/commit/6d6508765514c7c10719478a0430f5e47c9a96ac
msg367637 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 09:42
New changeset 4db245ee9ddbe6c53d375de59a35ff59dea2a8e0 by Pablo Galindo in branch 'master':
bpo-40334: refactor and cleanup for the PEG generators (GH-19775)
https://github.com/python/cpython/commit/4db245ee9ddbe6c53d375de59a35ff59dea2a8e0
msg367713 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 22:53
New changeset 69e802ed812e38cb68a4ab74af64b4f719b6cc78 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix test_peg_parser to actually use the old parser (GH-19778)
https://github.com/python/cpython/commit/69e802ed812e38cb68a4ab74af64b4f719b6cc78
msg367772 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-30 19:12
New changeset c001c09e9059ba04bc088349cd87a1374dc0754f by Guido van Rossum in branch 'master':
bpo-40334: Support type comments (GH-19780)
https://github.com/python/cpython/commit/c001c09e9059ba04bc088349cd87a1374dc0754f
msg367815 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-01 00:00
Warnings found by pyflakes:

Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:115:31 undefined name 'lines'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:116:25 undefined name 'lines'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:527:42 undefined name 'stmt'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:527:48 undefined name 'blocks'
msg367822 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-01 03:27
New changeset 3e0a6f37dfdd595be737baae00ec0e036a912615 by Lysandros Nikolaou in branch 'master':
bpo-40334: Add support for feature_version in new PEG parser (GH-19827)
https://github.com/python/cpython/commit/3e0a6f37dfdd595be737baae00ec0e036a912615
msg367835 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 10:12
> Warnings found by pyflakes:

Hummmm, "Lib/test/test_tools/test_c_analyzer/test_parser" has nothing to do with this PR. This looks like Eric Snow's c analyzer for avoiding globals and other stuff for sub interpreters.
msg367836 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 11:32
New changeset b796b3fb48283412d3caf52323c69690e5818d3d by Pablo Galindo in branch 'master':
bpo-40334: Simplify type handling in the PEG c_generator (GH-19818)
https://github.com/python/cpython/commit/b796b3fb48283412d3caf52323c69690e5818d3d
msg367839 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-01 12:43
While PR 19786 makes the PEG generator working on 3.6, it fails on 3.7. Seems there are more dependencies on wrong tokenizer. Please don't forget to fix this.
msg367840 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-01 12:46
>  Hummmm, "Lib/test/test_tools/test_c_analyzer/test_parser" has nothing to do with this PR. This looks like Eric Snow's c analyzer for avoiding globals and other stuff for sub interpreters.

Oh sorry. I analyzed dozens of pyflakes warning. When I saw "parser", I expected the issue to be new and so introduced by this bpo. Sorry about that ;-)
msg367843 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 13:13
New changeset 76c1b4d5c5a610c09943e1ee7ae18f1957804730 by Batuhan Taskaya in branch 'master':
bpo-40334: Improve column offsets for thrown syntax errors by Pegen (GH-19782)
https://github.com/python/cpython/commit/76c1b4d5c5a610c09943e1ee7ae18f1957804730
msg367845 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 13:45
> Seems there are more dependencies on wrong tokenizer. Please don't forget to fix this.

Thanks, Serhiy, will submit a PR soon!
msg367850 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-01 15:02
New changeset ea7297cf8f1aad4df8921a3d81a75118511afe77 by Pablo Galindo in branch 'master':
bpo-40334: unskip test_function_type in test_unparse with the new parser (GH-19837)
https://github.com/python/cpython/commit/ea7297cf8f1aad4df8921a3d81a75118511afe77
msg367854 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-01 15:32
New changeset d955241469c18c946924dba79c18a9ef200391ad by Pablo Galindo in branch 'master':
bpo-40334: Correct return value of func_type_comment (GH-19833)
https://github.com/python/cpython/commit/d955241469c18c946924dba79c18a9ef200391ad
msg367862 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 16:42
New changeset 3941d9700b2a272689cb8a8435b5c60a1466ef79 by Guido van Rossum in branch 'master':
bpo-40334: Refactor lambda_parameters similar to parameters (GH-19830)
https://github.com/python/cpython/commit/3941d9700b2a272689cb8a8435b5c60a1466ef79
msg367866 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 17:30
New changeset 03b7642265e65f198682f22648dbe6cf4fff9835 by Lysandros Nikolaou in branch 'master':
bpo-40334: Make the PyPegen* and PyParser* APIs more consistent (GH-19839)
https://github.com/python/cpython/commit/03b7642265e65f198682f22648dbe6cf4fff9835
msg367891 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 22:14
New changeset 7ba08ff7b41911f972d0750e068a2270e0dbd68f by Pablo Galindo in branch 'master':
bpo-40334: use the TOKENS file when checking dangling rules (GH-19849)
https://github.com/python/cpython/commit/7ba08ff7b41911f972d0750e068a2270e0dbd68f
msg367914 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-02 04:23
New changeset d2baff4301387e232495491f7291903cc1217d21 by Pablo Galindo in branch 'master':
bpo-40334: regenerate metaparser as part of regen-all (GH-19854)
https://github.com/python/cpython/commit/d2baff4301387e232495491f7291903cc1217d21
msg367916 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-02 05:06
Easy issue: the Python.gram file still has a comment saying something like

# Simplified grammar
msg367974 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-03 15:29
We also get:

  File "/usr/lib/python3.9/site-packages/dnf/comps.py", line 111
    return'.'.join(lcl)
         ^
SyntaxError: invalid string prefix
msg367976 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-05-03 15:37
This isn't related to this issue, but to bpo-40246. Please report it there as well.
msg367978 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-03 15:48
Ah, sorry, I got it mixed up. Reported there.
msg368004 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-04 00:20
New changeset 7f06af684a1882fdb19d20650825948b1d7996e5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Set error_indicator in _PyPegen_raise_error (GH-19887)
https://github.com/python/cpython/commit/7f06af684a1882fdb19d20650825948b1d7996e5
msg368034 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-04 10:58
New changeset e10e7c771bf06112c4a311e0ef6b8af6423b0cca by Lysandros Nikolaou in branch 'master':
bpo-40334: Spacialized error message for invalid args after bare '*' (GH-19865)
https://github.com/python/cpython/commit/e10e7c771bf06112c4a311e0ef6b8af6423b0cca
msg368288 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-06 18:11
New changeset 999ec9ab6af536cc2666a0847ec02331aaf00416 by Lysandros Nikolaou in branch 'master':
bpo-40334: Add type to the assignment rule in the grammar file (GH-19963)
https://github.com/python/cpython/commit/999ec9ab6af536cc2666a0847ec02331aaf00416
msg368302 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-06 21:54
New changeset 99db2a1db7a9b468a0ce8377d579f78fa03a2a34 by Pablo Galindo in branch 'master':
bpo-40334: Allow trailing comma in parenthesised context managers (GH-19964)
https://github.com/python/cpython/commit/99db2a1db7a9b468a0ce8377d579f78fa03a2a34
msg368304 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-06 22:14
New changeset 470aac4d8e76556bd8f820f3f3928dca2b4d2849 by Pablo Galindo in branch 'master':
bpo-40334: Generate comments in the parser code to improve debugging (GH-19966)
https://github.com/python/cpython/commit/470aac4d8e76556bd8f820f3f3928dca2b4d2849
msg368329 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-07 10:37
New changeset 2f37c355ab0e9ec9c1753985d27c41fa0bd719b9 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix error location upon parsing an invalid string literal (GH-19962)
https://github.com/python/cpython/commit/2f37c355ab0e9ec9c1753985d27c41fa0bd719b9
msg368330 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-07 10:44
New changeset 4638c6429575bd6de26b12b2af5df74d6568b553 by Lysandros Nikolaou in branch 'master':
bpo-40334: Error message for invalid default args in function call (GH-19973)
https://github.com/python/cpython/commit/4638c6429575bd6de26b12b2af5df74d6568b553
msg368566 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-10 04:34
New changeset ac7a92cc0a821699df48bc2e30a02c25d6338f78 by Pablo Galindo in branch 'master':
bpo-40334: Avoid collisions between parser variables and grammar variables (GH-19987)
https://github.com/python/cpython/commit/ac7a92cc0a821699df48bc2e30a02c25d6338f78
msg368567 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-10 04:40
FWIW I propose that we add another Misc/NEWS for the same issue summarizing what happened between alpha6 and beta1. (A lot!)
msg368593 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-10 22:50
There is some discrepancy with the codeop module when running with the new parser:

./python -c "import codeop; codeop.CommandCompiler()('raise = 2\n\n', symbol='exec')"

(No error)

./python -Xoldparser -c "import codeop; codeop.CommandCompiler()('raise = 2\n\n', symbol='exec')"
...
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax
msg368595 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-05-10 23:07
Thanks Pablo for replying to my tweet, a couple of other non-failing case in 
https://bugs.python.org/issue40585 that I filled concurrently to your message.

(also using this occasion to thanks all of you for your work on the new parser)
msg368598 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-05-10 23:44
Seem to not be in the new parser but simply in codeop in particular `def _maybe_compile`

The logic seem weird (but weird logic usually have a reason), it try to compile thrice by appending many `\n`.

1) Why do that and not return the first successful compile directly ? I'm not sure. 
2) It does compare the repr of error when  compile(source + "\n") and compile(source+'\n\n') and only raise if both are identical (which they are not in the new parser, they differ by `\n`...)

SyntaxError('invalid syntax', ('<input>', 1, 6, 'def a-b')) 
vs
SyntaxError('invalid syntax', ('<input>', 1, 6, 'def a-b\n'))


This logic seem to go back to the 2000s.
msg368599 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-11 00:10
It definitely has its reasons -- there are pernicious edge cases. Probably revealed by the tests.
msg368663 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-11 21:54
New changeset 27c0d9b54abaa4112d5a317b8aa78b39ad60a808 by Shantanu in branch 'master':
bpo-40334: produce specialized errors for invalid del targets (GH-19911)
https://github.com/python/cpython/commit/27c0d9b54abaa4112d5a317b8aa78b39ad60a808
msg368796 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-13 19:36
New changeset a15c9b3a0524e5ca0434d2ad11076677824af941 by Lysandros Nikolaou in branch 'master':
bpo-40334: Always show the caret on SyntaxErrors (GH-20050)
https://github.com/python/cpython/commit/a15c9b3a0524e5ca0434d2ad11076677824af941
msg368892 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-15 01:04
New changeset 16ab07063cb564c1937714bd39d6915172f005b5 by Pablo Galindo in branch 'master':
bpo-40334: Correctly identify invalid target in assignment errors (GH-20076)
https://github.com/python/cpython/commit/16ab07063cb564c1937714bd39d6915172f005b5
msg369090 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-17 03:19
New changeset 2c8cd06afe8e0abb52367f85978f19b88e2df53e by Lysandros Nikolaou in branch 'master':
bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)
https://github.com/python/cpython/commit/2c8cd06afe8e0abb52367f85978f19b88e2df53e
msg369166 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-05-18 03:36
One minor nit:  We're getting a nuisance compiler warning:

test_peg_generator passed (2 min 40 sec)
/var/folders/1b/3499qp8s7xv5w0fvjcgl45100000gn/T/tmp90rifd5z/parse.c:97:19: warning: code will never be executed [-Wunreachable-code]
        p->mark = _mark;
                  ^~~~~
1 warning generated.
msg369203 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-05-18 10:32
I'm not getting any compiler warnings on macOS (clang) and on Ubuntu (gcc) and I can't find any related warnings on the Windows Github Actions logs either. Could you maybe post a verbose log of test_peg_generator, Raymond?
msg369288 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-18 19:14
New changeset 75b863aa97016c6813709eb620c43295f84dd51f by Lysandros Nikolaou in branch 'master':
bpo-40334: Reproduce error message for type comments on bare '*' in the new parser (GH-20151)
https://github.com/python/cpython/commit/75b863aa97016c6813709eb620c43295f84dd51f
msg369536 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-21 19:57
New changeset f50516e6a978ee694232512399dd1ab47aaebab1 by Batuhan Taskaya in branch 'master':
bpo-40334: Correctly generate C parser when assigned var is None (GH-20296)
https://github.com/python/cpython/commit/f50516e6a978ee694232512399dd1ab47aaebab1
msg369553 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-22 00:56
New changeset ae145833025b0156ee2a28219e3370f3b27b2a36 by Lysandros Nikolaou in branch 'master':
bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
https://github.com/python/cpython/commit/ae145833025b0156ee2a28219e3370f3b27b2a36
msg369554 - (view) Author: miss-islington (miss-islington) Date: 2020-05-22 01:15
New changeset 55c89235247d9dbe8a4463c9c64edc7e48826a44 by Miss Islington (bot) in branch '3.9':
bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
https://github.com/python/cpython/commit/55c89235247d9dbe8a4463c9c64edc7e48826a44
msg369837 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-24 22:20
New changeset cba503151056b448b7a3730dc36ef6655550ade5 by Batuhan Taskaya in branch 'master':
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
https://github.com/python/cpython/commit/cba503151056b448b7a3730dc36ef6655550ade5
msg369840 - (view) Author: miss-islington (miss-islington) Date: 2020-05-24 22:38
New changeset 82c274e3ba7d011e93805f1552e90baea1752cf1 by Miss Islington (bot) in branch '3.9':
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
https://github.com/python/cpython/commit/82c274e3ba7d011e93805f1552e90baea1752cf1
msg371841 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-18 23:10
New changeset 01ece63d42b830df106948db0aefa6c1ba24416a by Lysandros Nikolaou in branch 'master':
bpo-40334: Produce better error messages on invalid targets (GH-20106)
https://github.com/python/cpython/commit/01ece63d42b830df106948db0aefa6c1ba24416a
msg371842 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-19 00:04
New changeset a5442b26f46f1073d1eb78895d554be520105ecb by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40334: Produce better error messages on invalid targets (GH-20106) (GH-20973)
https://github.com/python/cpython/commit/a5442b26f46f1073d1eb78895d554be520105ecb
msg371848 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-19 07:54
All parts of the implementation are now in. New issues should be opened for any potential bugs.
History Date User Action Args 2022-04-11 14:59:29adminsetgithub: 84514 2020-06-19 07:54:58lys.nikolaousetstatus: open -> closed
resolution: fixed
messages: + msg371848

stage: patch review -> resolved

2020-06-19 00:04:05pablogsalsetmessages: + msg371842 2020-06-18 23:31:17lys.nikolaousetpull_requests: + pull_request20150 2020-06-18 23:10:50pablogsalsetmessages: + msg371841 2020-05-24 22:38:51miss-islingtonsetmessages: + msg369840 2020-05-24 22:20:31miss-islingtonsetpull_requests: + pull_request19632 2020-05-24 22:20:21pablogsalsetmessages: + msg369837 2020-05-24 21:53:59BTaskayasetpull_requests: + pull_request19631 2020-05-22 01:15:02miss-islingtonsetmessages: + msg369554 2020-05-22 00:57:05miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19578
2020-05-22 00:56:59pablogsalsetmessages: + msg369553 2020-05-21 19:57:56pablogsalsetmessages: + msg369536 2020-05-21 18:51:19BTaskayasetpull_requests: + pull_request19571 2020-05-18 19:14:53pablogsalsetmessages: + msg369288 2020-05-18 10:32:00lys.nikolaousetmessages: + msg369203 2020-05-18 03:36:21rhettingersetnosy: + rhettinger
messages: + msg369166
2020-05-17 15:28:06lys.nikolaousetpull_requests: + pull_request19456 2020-05-17 11:50:38lys.nikolaousetpull_requests: + pull_request19454 2020-05-17 03:19:27pablogsalsetmessages: + msg369090 2020-05-15 13:03:38lys.nikolaousetpull_requests: + pull_request19413 2020-05-15 01:04:57pablogsalsetmessages: + msg368892 2020-05-13 22:23:32pablogsalsetpull_requests: + pull_request19382 2020-05-13 19:36:39pablogsalsetmessages: + msg368796 2020-05-12 01:29:33lys.nikolaousetpull_requests: + pull_request19359 2020-05-11 22:11:40pablogsalsetpull_requests: - pull_request19349 2020-05-11 21:54:16gvanrossumsetmessages: + msg368663 2020-05-11 19:32:41pablogsalsetpull_requests: + pull_request19349 2020-05-11 00:10:29gvanrossumsetmessages: + msg368599 2020-05-10 23:44:03mbussonnsetmessages: + msg368598 2020-05-10 23:07:47mbussonnsetnosy: + mbussonn
messages: + msg368595
2020-05-10 22:50:12pablogsalsetmessages: + msg368593 2020-05-10 04:40:45gvanrossumsetmessages: + msg368567 2020-05-10 04:34:57gvanrossumsetmessages: + msg368566 2020-05-10 00:37:19pablogsalsetpull_requests: + pull_request19331 2020-05-08 15:08:54lys.nikolaousetpull_requests: + pull_request19315 2020-05-08 02:45:05pablogsalsetpull_requests: + pull_request19303 2020-05-07 10:44:10pablogsalsetmessages: + msg368330 2020-05-07 10:37:58pablogsalsetmessages: + msg368329 2020-05-07 00:33:02lys.nikolaousetpull_requests: + pull_request19290 2020-05-06 22:14:49pablogsalsetmessages: + msg368304 2020-05-06 21:54:42pablogsalsetmessages: + msg368302 2020-05-06 21:00:17pablogsalsetpull_requests: + pull_request19285 2020-05-06 18:47:00pablogsalsetpull_requests: + pull_request19281 2020-05-06 18:11:14pablogsalsetmessages: + msg368288 2020-05-06 17:52:03pablogsalsetpull_requests: + pull_request19280 2020-05-06 17:42:48lys.nikolaousetpull_requests: + pull_request19279 2020-05-06 16:19:26lys.nikolaousetpull_requests: + pull_request19278 2020-05-04 21:10:35hauntsaninjasetnosy: + hauntsaninja
pull_requests: + pull_request19226
2020-05-04 16:28:53vstinnersetnosy: - vstinner
2020-05-04 11:19:45hroncoksetnosy: - hroncok
2020-05-04 11:11:45cstrataksetnosy: - cstratak
2020-05-04 10:58:42pablogsalsetmessages: + msg368034 2020-05-04 00:20:12pablogsalsetmessages: + msg368004 2020-05-03 20:21:23lys.nikolaousetpull_requests: + pull_request19197 2020-05-03 15:48:15hroncoksetnosy: gvanrossum, vstinner, ned.deily, serhiy.storchaka, Anthony Sottile, cstratak, hroncok, lys.nikolaou, pablogsal, BTaskaya
messages: + msg367978 2020-05-03 15:37:11lys.nikolaousetmessages: + msg367976 2020-05-03 15:29:33hroncoksetnosy: + hroncok
messages: + msg367974
2020-05-02 20:11:56lys.nikolaousettype: resource usage -> enhancement 2020-05-02 18:38:12lys.nikolaousetpull_requests: + pull_request19179 2020-05-02 05:06:47gvanrossumsetmessages: + msg367916 2020-05-02 04:23:47pablogsalsetmessages: + msg367914 2020-05-02 03:56:27pablogsalsetpull_requests: + pull_request19170 2020-05-01 22:14:15pablogsalsetmessages: + msg367891 2020-05-01 21:26:45pablogsalsetpull_requests: + pull_request19166 2020-05-01 17:30:59pablogsalsetmessages: + msg367866 2020-05-01 16:42:10pablogsalsetmessages: + msg367862 2020-05-01 16:32:58lys.nikolaousetpull_requests: + pull_request19157 2020-05-01 15:32:16gvanrossumsetmessages: + msg367854 2020-05-01 15:02:13gvanrossumsetmessages: + msg367850 2020-05-01 14:34:28pablogsalsetpull_requests: + pull_request19155 2020-05-01 13:45:59pablogsalsetmessages: + msg367845 2020-05-01 13:13:55pablogsalsetmessages: + msg367843 2020-05-01 12:46:51vstinnersetmessages: + msg367840 2020-05-01 12:43:17serhiy.storchakasetmessages: + msg367839 2020-05-01 11:32:30pablogsalsetmessages: + msg367836 2020-05-01 10:55:18pablogsalsetpull_requests: + pull_request19152 2020-05-01 10:12:02pablogsalsetmessages: + msg367835 2020-05-01 03:27:59gvanrossumsetmessages: + msg367822 2020-05-01 02:57:51gvanrossumsetpull_requests: + pull_request19150 2020-05-01 02:42:47gvanrossumsetpull_requests: + pull_request19148 2020-05-01 00:33:07lys.nikolaousetpull_requests: + pull_request19147 2020-05-01 00:00:42vstinnersetnosy: + vstinner
messages: + msg367815
2020-04-30 19:12:32gvanrossumsetmessages: + msg367772 2020-04-30 17:23:04pablogsalsetpull_requests: + pull_request19138 2020-04-29 22:53:36pablogsalsetmessages: + msg367713 2020-04-29 09:42:31pablogsalsetmessages: + msg367637 2020-04-29 07:22:35serhiy.storchakasetpull_requests: + pull_request19109 2020-04-29 01:42:30pablogsalsetmessages: + msg367610 2020-04-29 01:35:35vstinnersetnosy: - vstinner
2020-04-29 01:34:47BTaskayasetpull_requests: + pull_request19104 2020-04-29 01:04:09pablogsalsetmessages: + msg367602 2020-04-29 00:43:54pablogsalsetmessages: + msg367598 2020-04-29 00:18:23gvanrossumsetpull_requests: + pull_request19100 2020-04-29 00:00:42pablogsalsetmessages: + msg367582 2020-04-28 23:58:07vstinnersetmessages: + msg367580 2020-04-28 23:55:34pablogsalsetpull_requests: + pull_request19099 2020-04-28 23:45:33vstinnersetmessages: + msg367577 2020-04-28 23:43:20pablogsalsetmessages: + msg367576 2020-04-28 23:36:53lys.nikolaousetpull_requests: + pull_request19098 2020-04-28 23:23:58pablogsalsetpull_requests: + pull_request19095 2020-04-28 23:07:01lys.nikolaousetpull_requests: + pull_request19094 2020-04-28 21:34:08lys.nikolaousetpull_requests: + pull_request19091 2020-04-28 21:27:47vstinnersetnosy: + vstinner
messages: + msg367564
2020-04-28 12:47:03vstinnersetnosy: - vstinner
2020-04-28 12:12:05pablogsalsetmessages: + msg367521 2020-04-28 00:26:58phrsetnosy: - phr
2020-04-28 00:24:54pablogsalsetmessages: + msg367474 2020-04-28 00:23:39pablogsalsetmessages: + msg367473 2020-04-28 00:15:11pablogsalsetpull_requests: + pull_request19067 2020-04-27 23:55:52gvanrossumsetmessages: + msg367471 2020-04-27 23:52:13phrsetnosy: + phr
messages: + msg367470
2020-04-27 22:58:04lys.nikolaousetpull_requests: + pull_request19066 2020-04-27 22:45:47lys.nikolaousetpull_requests: + pull_request19065 2020-04-27 22:15:32pablogsalsetmessages: + msg367467 2020-04-27 19:21:03serhiy.storchakasetmessages: + msg367448 2020-04-27 17:36:04pablogsalsetmessages: + msg367440 2020-04-27 17:13:52pablogsalsetpull_requests: + pull_request19058 2020-04-27 17:02:14pablogsalsetmessages: + msg367439 2020-04-26 22:42:26pablogsalsetpull_requests: + pull_request19045 2020-04-26 20:36:06pablogsalsetpull_requests: + pull_request19043 2020-04-25 00:45:55gvanrossumsetmessages: + msg367246 2020-04-25 00:20:03gvanrossumsetmessages: + msg367243 2020-04-24 23:09:21Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg367237
2020-04-24 18:26:50gvanrossumsetpull_requests: + pull_request19022 2020-04-24 14:56:57vstinnersetmessages: + msg367196 2020-04-24 13:53:46lys.nikolaousetmessages: + msg367191 2020-04-24 13:51:15pablogsalsetmessages: + msg367190 2020-04-24 03:56:20gvanrossumsetmessages: + msg367171 2020-04-24 02:33:26vstinnersetmessages: + msg367170 2020-04-23 23:53:36pablogsalsetmessages: + msg367160 2020-04-23 23:09:36gvanrossumsetpull_requests: + pull_request19015 2020-04-23 23:05:04lys.nikolaousetpull_requests: + pull_request19014 2020-04-23 22:57:52vstinnersetmessages: + msg367158 2020-04-23 22:43:02gvanrossumsetmessages: + msg367155 2020-04-23 22:21:14gvanrossumsetpull_requests: + pull_request19012 2020-04-23 19:06:44pablogsalsetpull_requests: + pull_request19004 2020-04-23 18:49:34pablogsalsetmessages: + msg367142 2020-04-23 18:48:31pablogsalsetmessages: + msg367141 2020-04-23 18:42:14serhiy.storchakasetmessages: + msg367140 2020-04-23 18:38:08pablogsalsetmessages: + msg367139 2020-04-23 18:13:34serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg367137
2020-04-23 18:12:04lys.nikolaousetmessages: + msg367136 2020-04-23 18:06:49vstinnersetmessages: + msg367135 2020-04-23 15:36:14pablogsalsetmessages: + msg367126 2020-04-23 15:17:09pablogsalsetmessages: + msg367121 2020-04-23 14:28:55vstinnersetmessages: + msg367115 2020-04-23 14:17:18cstrataksetnosy: + cstratak
messages: + msg367114
2020-04-23 14:10:50pablogsalsetmessages: + msg367113 2020-04-23 14:05:48vstinnersetmessages: + msg367112 2020-04-23 12:22:35pablogsalsetmessages: + msg367105 2020-04-23 12:00:37lys.nikolaousetpull_requests: + pull_request18999 2020-04-23 11:42:17pablogsalsetmessages: + msg367097 2020-04-23 06:52:42ned.deilysetmessages: + msg367084 2020-04-23 02:52:25pablogsalsetmessages: + msg367076 2020-04-23 02:51:15pablogsalsetmessages: + msg367075 2020-04-23 02:43:11pablogsalsetmessages: + msg367073 2020-04-23 02:23:22pablogsalsetpull_requests: + pull_request18996 2020-04-23 01:55:43ned.deilysetmessages: + msg367071 2020-04-23 01:34:56pablogsalsetpull_requests: + pull_request18994 2020-04-23 01:33:17Wade Sanchezsettype: resource usage 2020-04-23 01:18:51lys.nikolaousetmessages: + msg367069 2020-04-23 01:15:55vstinnersetmessages: + msg367068 2020-04-23 01:09:20vstinnersetmessages: + msg367067 2020-04-23 01:03:27vstinnersetmessages: + msg367066 2020-04-23 00:38:15pablogsalsetmessages: + msg367064 2020-04-23 00:12:19pablogsalsetpull_requests: + pull_request18993 2020-04-23 00:07:49vstinnersetpull_requests: + pull_request18992 2020-04-23 00:07:29lys.nikolaousetpull_requests: + pull_request18991 2020-04-22 23:42:52pablogsalsetmessages: + msg367063 2020-04-22 23:42:28pablogsalsetpull_requests: + pull_request18990 2020-04-22 23:21:25ned.deilysetnosy: + ned.deily
messages: + msg367061
2020-04-22 23:19:28vstinnersetmessages: + msg367060 2020-04-22 23:13:52pablogsalsetmessages: + msg367059 2020-04-22 22:54:01pablogsalsetpull_requests: + pull_request18989 2020-04-22 22:34:23pablogsalsetpull_requests: + pull_request18988 2020-04-22 22:29:49pablogsalsetmessages: + msg367057 2020-04-22 21:07:54gvanrossumsetmessages: + msg367050 2020-04-22 15:16:53vstinnersetnosy: + vstinner
messages: + msg367009
2020-04-19 22:11:59lys.nikolaousetnosy: + lys.nikolaou
2020-04-19 20:21:23pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request18943 2020-04-19 20:17:57BTaskayasetnosy: + BTaskaya
2020-04-19 20:16:17gvanrossumcreate