Issue 44446: linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

demo:

import traceback
import io

async def foo():
    yield 1
    traceback.print_stack(file=io.StringIO())
    yield 2

async def bar():
    return [chunk async for chunk in foo()]


next(bar().__await__(), None)
print("working!")


Traceback (most recent call last):
  File "/home/graingert/projects/anyio/foo.py", line 13, in <module>
    next(bar().__await__(), None)
  File "/home/graingert/projects/anyio/foo.py", line 10, in bar
    return [chunk async for chunk in foo()]
  File "/home/graingert/projects/anyio/foo.py", line -1, in <listcomp>
  File "/home/graingert/projects/anyio/foo.py", line 6, in foo
    traceback.print_stack(file=io.StringIO())
  File "/usr/lib/python3.10/traceback.py", line 203, in print_stack
    print_list(extract_stack(f, limit=limit), file=file)
  File "/usr/lib/python3.10/traceback.py", line 224, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/usr/lib/python3.10/traceback.py", line 379, in extract
    f.line
  File "/usr/lib/python3.10/traceback.py", line 301, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/lib/python3.10/linecache.py", line 31, in getline
    if 1 <= lineno <= len(lines):
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

I bissected this to 088a15c49d99ecb4c3bef93f8f40dd513c6cae3b and submitted a patch making traceback.FrameSummary take into consideration that lineno might be None, I believe this is probably the correct fix.

Upon further investigation, there are actually two issues here. The first would be the one I identified already, traceback.FrameSummary not being prepared for lineno being None, but there is also a regression in lineno being invalid in this situation in the first place.

With only GH-26781, the traceback will look like the following:


  File "/home/anubis/git/cpython/rep.py", line 13, in <module>
    next(bar().__await__(), None)
  File "/home/anubis/git/cpython/rep.py", line 10, in bar
    return [chunk async for chunk in foo()]
  File "/home/anubis/git/cpython/rep.py", line None, in <listcomp>
  File "/home/anubis/git/cpython/rep.py", line 6, in foo
    traceback.print_stack()
working!


which is different from 3.9



  File "/home/anubis/git/cpython/rep.py", line 13, in <module>
    next(bar().__await__(), None)
  File "/home/anubis/git/cpython/rep.py", line 10, in bar
    return [chunk async for chunk in foo()]
  File "/home/anubis/git/cpython/rep.py", line 10, in <listcomp>
    return [chunk async for chunk in foo()]
  File "/home/anubis/git/cpython/rep.py", line 6, in foo
    traceback.print_stack()
working!


I bisected the second issue to b37181e69209746adc2119c471599a1ea5faa6c8 which moves generators to bytecode, and when doing so changes the behavior to set lineno to -1. I have opened a GH-26782 to fixing this.

Mark, can you take a look at this?

This appears to be a duplicate of https://bugs.python.org/issue44297

With the latest 3.10, I get:

  File "/home/mark/test/test.py", line 13, in <module>
    next(bar().__await__(), None)
  File "/home/mark/test/test.py", line 10, in bar
    return [chunk async for chunk in foo()]
  File "/home/mark/test/test.py", line 10, in <listcomp>
    return [chunk async for chunk in foo()]
  File "/home/mark/test/test.py", line 6, in foo
    traceback.print_stack()
working!


Thomas, can you confirm?

Beta 4 is in a few days. Can someone confirm of this is fixed or if it still needs a patch?

The issue that made the line number be missing is fixed, but GH-26781 is needed to account for 088a15c49d99ecb4c3bef93f8f40dd513c6cae3b, even though it is no longer triggered in this situation.

New changeset 91a8f8c16ca9a7e2466a8241d9b41769ef97d094 by Filipe Laíns in branch 'main':
bpo-44446: support lineno being None in traceback.FrameSummary (GH-26781)
https://github.com/python/cpython/commit/91a8f8c16ca9a7e2466a8241d9b41769ef97d094

New changeset 61eb9b5dfd919ba5d1ec9f7df0137f2e6d196972 by Pablo Galindo in branch '3.10':
[3.10] bpo-44446: support lineno being None in traceback.FrameSummary (GH-26781) (GH-27072)
https://github.com/python/cpython/commit/61eb9b5dfd919ba5d1ec9f7df0137f2e6d196972

Feel free to reopen if we are missing anything

If I understand correctly this should be fixed? In which 3.10.* version should this be fixed?

The reason why I'm asking is that I ran into this issue when using Dask (2022.02.0) with multithreading on Python 3.10.2:

Exception in thread Profile:
Traceback (most recent call last):
  File "./lib/python3.10/site-packages/distributed/profile.py", line 115, in process
    d = state["children"][ident]
KeyError: '_all_objs;./lib/python3.10/site-packages/bokeh/embed/bundle.py;357'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "./lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "./lib/python3.10/site-packages/distributed/profile.py", line 274, in _watch
    process(frame, None, recent, omit=omit)
  File "./lib/python3.10/site-packages/distributed/profile.py", line 119, in process
    "description": info_frame(frame),
  File "./lib/python3.10/site-packages/distributed/profile.py", line 72, in info_frame
    line = linecache.getline(co.co_filename, frame.f_lineno, frame.f_globals).lstrip()
  File "./lib/python3.10/linecache.py", line 31, in getline
    if 1 <= lineno <= len(lines):
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

This was fixed in 3.10.0 if I am not mistaken. Could you provide a reproducer, please?

messages: + msg414346
versions: + Python 3.10, - Python 3.11

messages: + msg396101
versions: + Python 3.11, - Python 3.10

pull_requests: + pull_request25366
stage: patch review