Issue 25872: multithreading traceback KeyError when modifying file

Messages (12) msg256469 - (view) Author: Michael Allen (Michael Allen) Date: 2015-12-15 17:21

Modifying a file while getting a stacktrace across multiple threads causes linecache's cache to bust and del to be called on the global cache variable. This is not thread safe and raises a KeyError.

Reproducible with,

import threading
import traceback

def main():
    with open(__file__, 'a') as fp:
        fp.write(' ')
        traceback.format_stack()

threads = [
    threading.Thread(target=main)
    for i in range(100)
]
map(lambda t: t.start(), threads)
map(lambda t: t.join(), threads)

I see the following error,

Exception in thread Thread-56:
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test.py", line 7, in main
    traceback.format_stack()
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/traceback.py", line 279, in format_stack
    return format_list(extract_stack(f, limit))
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/traceback.py", line 305, in extract_stack
    linecache.checkcache(filename)
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/linecache.py", line 69, in checkcache
    del cache[filename]
KeyError: 'test.py'

Possible solution is to ignore KeyError on del cache[filename].

msg323180 - (view) Author: Karthikeyan Singaravelan (xtreak) * Date: 2018-08-06 07:41

It seems there was a major refactor in traceback module with 6bc2c1e7ebf359224e5e547f58ffc2c42cb36a39 where this was fixed in Python 3. Ignoring the KeyError seems reasonable to me.

Thanks

msg359390 - (view) Author: Cheryl Sabella (cheryl.sabella) * Date: 2020-01-06 00:27

As @xtreak said, this looks like it was fixed for Python 3 and was only an issue for 2.7, so I'm closing the issue.

msg359392 - (view) Author: Michael Graczyk (Michael Graczyk) * Date: 2020-01-06 00:50

This issue still exists in Python 3. The repro just needs to be changed so that the threads are actually started.

- map(lambda t: t.start(), threads)
- map(lambda t: t.join(), threads)
+ [t.start() for t in threads]
+ [t.join() for t in threads]

My fix is linked.

msg359427 - (view) Author: Karthikeyan Singaravelan (xtreak) * Date: 2020-01-06 14:38

Thanks Michael, reopening. I was wrong while trying the reproducer since map is lazy in Python 3 and threads were not executed.

msg370301 - (view) Author: miss-islington (miss-islington) Date: 2020-05-29 11:59

New changeset b86636bff4b29ce23c886df079715dd951f13a07 by Andrew Kuchling in branch '3.8':
[3.8] bpo-25872: Fix KeyError in linecache when multithreaded (GH-18007) (GH-20092)
https://github.com/python/cpython/commit/b86636bff4b29ce23c886df079715dd951f13a07

msg370302 - (view) Author: miss-islington (miss-islington) Date: 2020-05-29 12:17

New changeset 852e8a7ed4d3d48e5c1c8120cfc932eb6a84bb8e by Miss Islington (bot) in branch '3.7':
[3.8] bpo-25872: Fix KeyError in linecache when multithreaded (GH-18007) (GH-20092)
https://github.com/python/cpython/commit/852e8a7ed4d3d48e5c1c8120cfc932eb6a84bb8e

msg381409 - (view) Author: Irit Katriel (iritkatriel) * Date: 2020-11-19 12:52

The issue was fixed but a unit test for this still needs to be added.

msg393187 - (view) Author: So Ukiyama (uniocto) * Date: 2021-05-07 14:46

I apologize if this is rude, as I am not familiar with this method.
I created a following PR to add unit tests about this issue.

https://github.com/python/cpython/pull/25913

I would be happy to receive feedback on the PR.

msg393872 - (view) Author: Irit Katriel (iritkatriel) * Date: 2021-05-18 13:54

New changeset 373741a97c9f6ffee427c2b4eaccb74347af228a by Irit Katriel in branch '3.10':
[3.10] bpo-25872: Add unit tests for linecache and threading (GH-25913) (GH-26212)
https://github.com/python/cpython/commit/373741a97c9f6ffee427c2b4eaccb74347af228a

msg393874 - (view) Author: Irit Katriel (iritkatriel) * Date: 2021-05-18 14:26

New changeset c05d8a6b67785450b1fec0d30fe26d5478bc4f0b by Irit Katriel in branch '3.9':
bpo-25872: Add unit tests for linecache and threading (GH-25913) (GH-26211)
https://github.com/python/cpython/commit/c05d8a6b67785450b1fec0d30fe26d5478bc4f0b

msg393875 - (view) Author: Irit Katriel (iritkatriel) * Date: 2021-05-18 14:27

@uniocto - thank you for the tests.

History Date User Action Args 2022-04-11 14:58:25adminsetgithub: 70060 2021-05-18 14:27:03iritkatrielsetstatus: open -> closed
resolution: fixed
messages: + msg393875

stage: patch review -> resolved

2021-05-18 14:26:07iritkatrielsetmessages: + msg393874 2021-05-18 13:54:06iritkatrielsetmessages: + msg393872 2021-05-18 10:53:06iritkatrielsetcomponents: + Library (Lib), - Interpreter Core 2021-05-18 10:52:06iritkatrielsetpull_requests: + pull_request24829 2021-05-18 10:38:48iritkatrielsetpull_requests: + pull_request24828 2021-05-18 08:57:09miss-islingtonsetpull_requests: + pull_request24825 2021-05-07 14:46:44unioctosetnosy: + uniocto
messages: + msg393187
components: + Interpreter Core, - Library (Lib)
2021-05-05 09:39:14python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request24582
stage: test needed -> patch review

2020-11-19 12:52:17iritkatrielsetnosy: + iritkatriel
messages: + msg381409

keywords: + easy, - patch
stage: patch review -> test needed

2020-05-29 12:17:46miss-islingtonsetmessages: + msg370302 2020-05-29 11:59:57miss-islingtonsetpull_requests: + pull_request19755 2020-05-29 11:59:47miss-islingtonsetmessages: + msg370301 2020-05-18 16:49:34rahul-kumisetnosy: + rahul-kumi
2020-05-14 19:19:52akuchlingsetnosy: + akuchling
pull_requests: + pull_request19398
2020-05-14 16:37:08miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19394
2020-01-15 04:24:26Michael Graczyksetstage: patch review
pull_requests: + pull_request17408 2020-01-06 14:38:49xtreaksetstatus: closed -> open
versions: + Python 3.9, - Python 2.7
messages: + msg359427

resolution: wont fix -> (no value)
stage: resolved -> (no value)

2020-01-06 00:50:41Michael Graczyksetnosy: + Michael Graczyk
messages: + msg359392
2020-01-06 00:27:17cheryl.sabellasetstatus: open -> closed

nosy: + cheryl.sabella
messages: + msg359390

resolution: wont fix
stage: patch review -> resolved

2019-11-23 02:07:22python-devsetpull_requests: + pull_request16845 2019-02-27 00:40:02cunkelsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12085 2018-08-06 07:41:56xtreaksetnosy: + xtreak
messages: + msg323180
2018-07-11 07:41:59serhiy.storchakasettype: crash -> behavior 2015-12-15 17:21:04Michael Allencreate