◐ Shell
reader mode source ↗
Skip to content

bpo-31061: fix crash in asyncio speedup module#2966

Merged
methane merged 18 commits into
python:masterfrom
thehesiod-forks:thehesiod/aio_crash_fix
Aug 2, 2017
Merged

bpo-31061: fix crash in asyncio speedup module#2966
methane merged 18 commits into
python:masterfrom
thehesiod-forks:thehesiod/aio_crash_fix

Conversation

@thehesiod

@thehesiod thehesiod commented Aug 1, 2017

Copy link
Copy Markdown
Contributor

@1st1 1st1 requested a review from methane August 1, 2017 01:25
@1st1 1st1 self-assigned this Aug 1, 2017
@thehesiod thehesiod changed the title attempt to fix http://bugs.python.org/issue31061 Aug 1, 2017
@thehesiod

thehesiod commented Aug 1, 2017

Copy link
Copy Markdown
Contributor Author

this is my first cpython PR, let me know what else needs to be done to get this into any other releases it's required in.

@methane

methane commented Aug 1, 2017

Copy link
Copy Markdown
Member

Since _asinciomodule is compiled with python interpreter, it can use private macro APIs
(_PyObject_GC_TRACK and _PyObject_GC_UNTRACK).

@methane

methane commented Aug 1, 2017

Copy link
Copy Markdown
Member

Oh, sorry, macro version API broke Windows build and you used public APIs to fix it.

@methane

methane commented Aug 1, 2017

Copy link
Copy Markdown
Member

@thehesiod

Copy link
Copy Markdown
Contributor Author

added news and reverted LRU change

@thehesiod

Copy link
Copy Markdown
Contributor Author

should we put as many fixes as we can in this PR, or create another PR with as many as we can? I need the defaultdict one too...and maybe even others as who knows what third party libraries do. I feel like we've opened a pandoras box of crashers :)

@methane

methane commented Aug 1, 2017

Copy link
Copy Markdown
Member

I already created #2974.
More pull request for updating document is welcome.

@methane methane added the type-bug An unexpected behavior, bug, or error label Aug 1, 2017
@methane methane changed the title attempt to fix bpo-31061 Aug 1, 2017
@1st1

1st1 commented Aug 1, 2017

Copy link
Copy Markdown
Member

The latest revision LGTM. But can you use blurb to generate the NEWS entry?

27 hidden items Load more…
@thehesiod

Copy link
Copy Markdown
Contributor Author

@1st1 should I remove the one in NEWS? Also was hoping I could add a testcase for Task but couldn't figure out how to repro it, especially given Task inherits from Future so I don't want to trigger the Future's crash

@1st1

1st1 commented Aug 1, 2017

Copy link
Copy Markdown
Member

@1st1 should I remove the one in NEWS?

Yes, please!

Also was hoping I could add a testcase for Task but couldn't figure out how to repro it, especially given Task inherits from Future so I don't want to trigger the Future's crash

I'm trying to come up with a test.

@1st1

1st1 commented Aug 1, 2017

Copy link
Copy Markdown
Member

The following test crashes unpatched CPython:

    import gc

    def test_task_del_collect(self):
        async def coro():
            return Evil()

        class Evil:
            def __del__(self):
                gc.collect()

        for i in range(100):
            task = self.new_task(self.loop, coro())
            self.loop.run_until_complete(task)

@1st1

1st1 commented Aug 1, 2017

Copy link
Copy Markdown
Member

The test also fails if we fix only Task, but not Future.

@thehesiod

Copy link
Copy Markdown
Contributor Author

so I had something similar but it doesn't crash:

import gc
import asyncio

class Evil:
    def __del__(self):
        gc.collect()


async def run():
    return Evil()


loop = asyncio.get_event_loop()

for i in range(100):
    loop.run_until_complete(asyncio.Task(run()))
#    f = asyncio.Future()
#    f.set_result(Evil())

@thehesiod

Copy link
Copy Markdown
Contributor Author

ah, figured it out, had to have multiple tasks pending in the run loop.

@thehesiod

thehesiod commented Aug 1, 2017

Copy link
Copy Markdown
Contributor Author

ya realized as I was going to work, this should fix it

@1st1 1st1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

I have no further comments, LGTM. @methane do you approve this PR?

@thehesiod

Copy link
Copy Markdown
Contributor Author

this set of PRs is a massive win for Python. It would be nice if a core dev could look and find a way to nip it at the bud to avoid these hacky calls, as shown they're highly error prone. Who knows how many third party modules are missing them too.

@methane methane left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM

@methane methane merged commit de34cbe into python:master Aug 2, 2017
methane pushed a commit to methane/cpython that referenced this pull request Aug 2, 2017
@bedevere-bot

Copy link
Copy Markdown

GH-2984 is a backport of this pull request to the 3.6 branch.

@1st1

1st1 commented Aug 2, 2017

Copy link
Copy Markdown
Member

Do we have C Task/Future in 3.5?

@methane

methane commented Aug 2, 2017

Copy link
Copy Markdown
Member

No. asyncio speedup is introduced at 3.6

@thehesiod thehesiod deleted the thehesiod/aio_crash_fix branch August 2, 2017 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type-bug An unexpected behavior, bug, or error

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants