Issue 35894: Apparent regression in 3.8-dev: 'TypeError: required field "type_ignores" missing from Module'

Travis provides a "3.8-dev" python, which is updated regularly to track cpython master. When running our tests on this Python, specifically version

  python: 3.8.0a0 (heads/master:f75d59e, Feb  3 2019, 07:27:24) 

we just started getting tracebacks:

  TypeError                                 Traceback (most recent call last)
  /opt/python/3.8-dev/lib/python3.8/codeop.py in __call__(self, source, filename, symbol)
      131 
      132     def __call__(self, source, filename, symbol):
  --> 133         codeob = compile(source, filename, symbol, self.flags, 1)
      134         for feature in _features:
      135             if codeob.co_flags & feature.compiler_flag:
  TypeError: required field "type_ignores" missing from Module

(Full log: https://travis-ci.org/python-trio/trio/jobs/488312057)

Grepping through git diffs for 'type_ignores' suggests that this is probably related to bpo-35766.

I haven't dug into this in detail, but it seems to be happening on tests using IPython. The lack of further traceback suggests to me that the exception is happening inside IPython's guts (it has some hacks to try to figure out which parts of the traceback are in user-defined code versus its own internal code, and tries to hide the latter when printing tracebacks). The crash is in codeop.Compile.__call__, and IPython does create ast.Module objects and pass them to codeop.Compile.__call__:

https://github.com/ipython/ipython/blob/512d47340c09d184e20811ca46aaa2f862bcbafe/IPython/core/interactiveshell.py#L3199-L3200

Maybe ast.Module needs to default-initialize the new type_ignores field, or compile() needs to be tolerant of it being missing?

Yeah, this definitely changed. When we updated typed_ast to Python 3.7 recently, the mypy project looked into making certain arguments to AST certain nodes optional, but the code responsible for those arguments is too weird to easily make that work (it's all generated), so we decided to leave it alone.

IMO the AST in general can't be considered a stable API like other stdlib APIs, since when the syntax changes, well, the AST changes, and sometimes that means that nodes change. For *consumers* of the AST we generally manage to keep things backwards compatible, but for producers, there just is no guarantee.

I think the best way forward is for your code to check the Python version and if it's >= 3.8, pass an empty list as the second argument to Module().

If you look at the changes to Python.asdl in the PR you found, you'll see what other nodes have changed.

Oh, that's not my code, it's the core of IPython's REPL :-). I just filed a bug with them, referencing this one: https://github.com/ipython/ipython/issues/11590

there's other optional fields in the ast, type ignores don't seem essential to the `Module`, could those be made optional as well?

> [T]here's other optional fields in the ast, type ignores don't seem essential to the `Module`, could those be made optional as well?

I think you're referring to the `?` syntax in `Python.asdl`.  But the `type_ignores` attribute is already a list (using `*`) and AFAICT you cannot combine `?` and `*`.  You have to provide an empty list.