Issue 40807: Codeop: Show warnings once during _maybe_compile

Messages (11) msg370208 - (view) Author: Cheryl Sabella (cheryl.sabella) * Date: 2020-05-28 13:55

When calling `codeop._maybe_compile`, `compile` is run three times.  If the code being compiled causes a warning message, the warning is generated each time that `compile` is called, thus (possibly) showing the message three times.


See msg370163 and that issue for context.

msg370722 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-04 19:05

The purpose of code and codeop is to imitate interactive python in python.  Calling compile() thrice is an implementation detail.  Exposing that is a bug.  On #37824, Serhiy reported 3 'IDLE' bugs for 3.7 to 3.9: tripled DeprecationWarning (now joined by SyntaxWarning), printing to console (if available) instead of Shell, and an exit exception.  This issue is continuation of the triplicate issue, given that the bug is not in IDLE.  As a bug, I think it should still be backported.

I looked more carefully at the behavior of the standard REPL with different statement inputs.  Before executing, if the statement is complete, it may raise a compile exception, usually SyntaxError, or emit one or more a warnings, once for each instance.  For multiline statements, DeprecationWarnings are emitted immediately after the line with the deprecated code while SyntaxWarnings are emitted, if there is no syntax error, just before execution.  SyntaxWarnings for a given slice of code are not repeated.  However, at least some of these details have to be enforced in the UI code that repeatedly calls compile or _maybe_compile as lines are added, not in compile, and should not be in _maybe_compile.  Both have no memory or previous calls.  (I don't think that IDLE necessarily *must* do the same either.)

Looking at compile itself, it appears that warnings are not emitted if there is an exception.  But I am puzzled at this in 3.9 and master.

>>> if '\e' is 1:
<stdin>:1: DeprecationWarning: invalid escape sequence \e
...
versus
>>> compile('if '\e' is 1:\n', '', 'single')
  File "<stdin>", line 1
SyntaxError: unexpected character after line continuation character

Something must cause "'\e'" to be a warning instead of an error.

This is enough to experiment with your patch applied.

msg370724 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-04 19:14

I am not sure if 3 compiles are always needed.  At one time, compile required a final '\n' to not raise, but that is no longer true.  And as near as I can tell, 'code without final newline\n' and 'code without final newline\n\n' always compile the same.  Certainly, Shell does not need to compile without a final newline.  But we can look at this in a separate issue.  If IDLE's requirement is different from code.II's, I believe it could pass its own compile function.

msg370738 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-04 23:40

New changeset 052d3fc0907be253cfd64b2c737a0b0aca586011 by Cheryl Sabella in branch 'master':
bpo-40807: Show warnings once from codeop._maybe_compile (#20486)
https://github.com/python/cpython/commit/052d3fc0907be253cfd64b2c737a0b0aca586011

msg370739 - (view) Author: miss-islington (miss-islington) Date: 2020-06-04 23:59

New changeset 3dfe549bfa08a099b88efd030669359199f31064 by Miss Islington (bot) in branch '3.9':
bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)
https://github.com/python/cpython/commit/3dfe549bfa08a099b88efd030669359199f31064

msg370748 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-05 02:42

New changeset a5d6aba318ead9cc756ba750a70da41f5def3f8f by Terry Jan Reedy in branch '3.8':
[3.8] bpo-40807: Backport test_codeop change [GH-19670]
https://github.com/python/cpython/commit/a5d6aba318ead9cc756ba750a70da41f5def3f8f

msg370750 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-05 03:00

New changeset 12d3061c7819a73d891dcce44327410eaf0e1bc2 by Miss Islington (bot) in branch '3.7':
[3.8] bpo-40807: Backport test_codeop change [GH-19670]
https://github.com/python/cpython/commit/12d3061c7819a73d891dcce44327410eaf0e1bc2

msg370751 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-05 03:01

This is a spinoff of #37824.

My above comments about compile are wrong.  The example nests '' within ''.  And, with freshly compiled master, I found an example where Deprecation warning, but not SyntaxWarning, accompanies SyntaxError, and which requires trailing \n.

>>> compile("if'\e' is 1: 1", '', 'single')
DeprecationWarning...
SyntaxError: unexpected EOF while parsing
>>> compile("if'\e' is 1: 1\n", '', 'single')
DeprecationWarning...
SyntaxWarning...
<code object ...>

Interactively,
>>> if'\e' is 1: 1
<stdin>:1: DeprecationWarning: invalid escape sequence \e
... 
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
>>>

In IDLE, DeprecationWarning is repeated, but that is another issue.

The secondary prompts seems like a bug,  The statement is complete and entering anything but a comment is a SyntaxError.  (A comment results in another ... prompt.)

msg370814 - (view) Author: miss-islington (miss-islington) Date: 2020-06-06 08:24

New changeset c067183605cf84bb1a246635f52827251d0476f8 by Miss Islington (bot) in branch '3.8':
bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)
https://github.com/python/cpython/commit/c067183605cf84bb1a246635f52827251d0476f8

msg371102 - (view) Author: Terry J. Reedy (terry.reedy) * Date: 2020-06-09 13:12

The 3.7 backport is needed but required a revised example.  The changeset is 4b378acb97a575892c0e372a6bb0c17da1ccdf3e.

PRs 20486, 20639, 20672, and 20673 are the main patch.  PRs 20646 and 20647 for 3.8 and 3.7 backport a test_codeop import change made elsewhere and not backported.  The alternative would have been changing our backport to the older naming convention.

msg375155 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-08-10 22:07

I think that might have introduce https://bugs.python.org/issue41520 where now `warnings.simplefilter('error', SyntaxWarning)` is silently ignored...

History Date User Action Args 2022-04-11 14:59:31adminsetgithub: 84984 2020-08-12 17:54:48vstinnersetnosy: + vstinner

pull_requests: + pull_request20976

2020-08-10 22:07:08mbussonnsetnosy: + mbussonn
messages: + msg375155
2020-06-09 13:12:26terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg371102

stage: patch review -> resolved

2020-06-06 08:24:48miss-islingtonsetmessages: + msg370814 2020-06-06 08:06:50miss-islingtonsetpull_requests: + pull_request19889 2020-06-06 08:06:43miss-islingtonsetpull_requests: + pull_request19888 2020-06-05 03:01:17terry.reedysetmessages: + msg370751 2020-06-05 03:00:58terry.reedysetmessages: + msg370750 2020-06-05 02:43:05miss-islingtonsetpull_requests: + pull_request19867 2020-06-05 02:42:51terry.reedysetmessages: + msg370748 2020-06-05 01:58:29terry.reedysetpull_requests: + pull_request19866 2020-06-04 23:59:51miss-islingtonsetmessages: + msg370739 2020-06-04 23:41:00miss-islingtonsetpull_requests: + pull_request19861 2020-06-04 23:40:51miss-islingtonsetpull_requests: + pull_request19860 2020-06-04 23:40:43miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19859
2020-06-04 23:40:28terry.reedysetmessages: + msg370738 2020-06-04 19:14:19terry.reedysetmessages: + msg370724 2020-06-04 19:05:12terry.reedysetnosy: + terry.reedy, serhiy.storchaka
title: CODEOP: Show warnings once during _maybe_compile -> Codeop: Show warnings once during _maybe_compile
messages: + msg370722

versions: + Python 3.7, Python 3.8
type: enhancement -> behavior

2020-05-28 13:57:54cheryl.sabellasetkeywords: + patch
stage: patch review
pull_requests: + pull_request19735 2020-05-28 13:55:08cheryl.sabellacreate