◐ Shell
reader mode source ↗
Skip to content

GH-66285: fix forking in asyncio#99539

Merged
kumaraditya303 merged 8 commits into
python:mainfrom
kumaraditya303:async-fork
Nov 24, 2022
Merged

GH-66285: fix forking in asyncio#99539
kumaraditya303 merged 8 commits into
python:mainfrom
kumaraditya303:async-fork

Conversation

@kumaraditya303

@kumaraditya303 kumaraditya303 commented Nov 16, 2022 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Contributor

@bedevere-bot bedevere-bot mentioned this pull request Nov 16, 2022
Closed
@kumaraditya303 kumaraditya303 marked this pull request as ready for review November 17, 2022 11:11
@kumaraditya303 kumaraditya303 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 17, 2022
@bedevere-bot

bedevere-bot commented Nov 17, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit d6ef1a6 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 17, 2022
@gvanrossum

gvanrossum commented Nov 18, 2022

Copy link
Copy Markdown
Member

See my comments on the issue, @kumaraditya303 .

@kumaraditya303

kumaraditya303 commented Nov 21, 2022

Copy link
Copy Markdown
Contributor Author

See my comments on the issue,

@gvanrossum: I have added more tests and also added a subprocess test.

@kumaraditya303 kumaraditya303 merged commit 0c1fbc1 into python:main Nov 24, 2022
@kumaraditya303 kumaraditya303 deleted the async-fork branch November 24, 2022 03:40
@bedevere-bot

bedevere-bot commented Nov 24, 2022

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x has failed when building commit 0c1fbc1.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/469/builds/3768) and take a look at the build logs.
  4. Check if the failure is related to this commit (0c1fbc1) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/469/builds/3768

Failed tests:

  • test_asyncio

Failed subtests:

  • test_fork_signal_handling - test.test_asyncio.test_unix_events.TestFork.test_fork_signal_handling
  • test_fork_asyncio_run - test.test_asyncio.test_unix_events.TestFork.test_fork_asyncio_run
  • test_fork_asyncio_subprocess - test.test_asyncio.test_unix_events.TestFork.test_fork_asyncio_subprocess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

416 tests OK.

10 slowest tests:

  • test_gdb: 6 min 6 sec
  • test_concurrent_futures: 2 min 46 sec
  • test_capi: 2 min 7 sec
  • test_asyncio: 1 min 57 sec
  • test_multiprocessing_spawn: 1 min 44 sec
  • test_multiprocessing_forkserver: 1 min 10 sec
  • test_multiprocessing_fork: 1 min 3 sec
  • test_tokenize: 59.6 sec
  • test_signal: 48.9 sec
  • test_socket: 46.8 sec

1 test failed:
test_asyncio

16 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_peg_generator test_startfile
test_tix test_tkinter test_ttk test_winconsoleio test_winreg
test_winsound test_wmi test_zipfile64

1 re-run test:
test_asyncio

Total duration: 10 min 8 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1953, in test_fork_asyncio_subprocess
    result = manager.Value('i', 1)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
    token, exp = self._create(typeid, *args, **kwds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
    digest = hmac.new(authkey, message, 'md5').digest()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in new
    return HMAC(key, msg, digestmod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: no reason supplied


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1902, in test_fork_signal_handling
    child_started = manager.Event()
                    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
    token, exp = self._create(typeid, *args, **kwds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
    digest = hmac.new(authkey, message, 'md5').digest()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in new
    return HMAC(key, msg, digestmod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: no reason supplied


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1937, in test_fork_asyncio_run
    result = manager.Value('i', 0)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
    token, exp = self._create(typeid, *args, **kwds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
    digest = hmac.new(authkey, message, 'md5').digest()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in new
    return HMAC(key, msg, digestmod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: no reason supplied

@bedevere-bot

bedevere-bot commented Nov 24, 2022

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 0c1fbc1.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/954) and take a look at the build logs.
  4. Check if the failure is related to this commit (0c1fbc1) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/954

Failed tests:

  • test_traceback

Failed subtests:

  • test_import_from_error_bad_suggestions_do_not_trigger_for_small_names - test.test_traceback.CPythonSuggestionFormattingTests.test_import_from_error_bad_suggestions_do_not_trigger_for_small_names

Summary of the results of the build (if available):

== Tests result: FAILURE ==

325 tests OK.

10 slowest tests:

  • test_tokenize: 51.3 sec
  • test_unparse: 42.7 sec
  • test_capi: 33.1 sec
  • test_lib2to3: 29.2 sec
  • test_unicodedata: 25.0 sec
  • test_decimal: 16.0 sec
  • test_statistics: 14.2 sec
  • test_pickle: 13.1 sec
  • test_zipfile: 9.4 sec
  • test_buffer: 7.5 sec

1 test failed:
test_traceback

107 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio test_bz2
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_gzip test_httplib test_httpservers test_idle
test_imaplib test_interpreters test_ioctl test_kqueue
test_launcher test_lzma test_mailbox test_mmap test_msilib
test_multiprocessing_fork test_multiprocessing_forkserver
test_multiprocessing_main_handling test_multiprocessing_spawn
test_nis test_openpty test_ossaudiodev test_pdb test_peg_generator
test_perf_profiler test_pipes test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_spwd test_sqlite3 test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib test_urllib2 test_urllib2_localnet test_urllib2net
test_urllib_response test_urllibnet test_venv test_wait3
test_wait4 test_webbrowser test_winconsoleio test_winreg
test_winsound test_wmi test_wsgiref test_xmlrpc test_xmlrpc_net
test_xxlimited test_zipfile64 test_zipimport_support test_zlib
test_zoneinfo
0:04:47 load avg: 2.78
0:04:47 load avg: 2.78 Re-running failed tests is not supported with --python host runner option.

Total duration: 4 min 47 sec

Click to see traceback logs 
TracebackFormatTests.test_format_stack) ... ok


TracebackFormatTests.test_exception_group_deep_recursion_traceback) ... ok


TracebackCases.test_format_exception_only_exc) ... ok


TracebackException.test_compact_no_cause) ... ok


TracebackCases.test_base_exception) ... ok


TracebackErrorCaretTests.test_specialization_variations) ... ok


TracebackException.test_locals) ... ok


TracebackFormatTests.test_print_exception_bad_type_python) ... ok


TracebackErrorCaretTests.test_caret_multiline_expression) ... ok


TracebackCases.test_print_traceback_at_exit) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_traceback_specialization_with_syntax_error) ... ok


TracebackException.test_compact_with_cause) ... ok


TracebackFormatTests.test_recursive_traceback_python) ... ok


TracebackException.test_long_context_chain) ... ok


TracebackException_ExceptionGroups.test_exception_group_format_exception_only) ... ok


TracebackFormatTests.test_print_exception_bad_type_capi) ... ok


TracebackCases.test_extract_stack) ... ok


TracebackCases.test_nocaret) ... ok


TracebackErrorCaretTests.test_multiline_method_call_a) ... ok


TracebackCases.test_no_caret_with_no_debug_ranges_flag) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_traceback_very_long_line) ... ok


TracebackErrorCaretTests.test_caret_exception_group) ... ok


TracebackErrorCaretTests.test_basic_caret) ... ok


Traceback (most recent call last):
  File "/Lib/test/test_traceback.py", line 3173, in test_import_from_error_bad_suggestions_do_not_trigger_for_small_names
    self.assertNotIn("mom", actual)
AssertionError: 'mom' unexpectedly found in "ImportError: cannot import name 'b' from 'nmQWXqKJcBDACXLN' (/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.wasi/build/build_oot/host/build/test_python_496508æ/test_python_worker_830271æ/tmp6fv8mom7/nmQWXqKJcBDACXLN.py)"


TracebackException.test_comparison_equivalent_exceptions_are_equal) ... ok


TracebackErrorCaretTests.test_multiline_method_call_c) ... ok


TracebackErrorCaretTests.test_caret_for_subscript_unicode) ... ok


TracebackCases.test_no_caret_with_no_debug_ranges_flag_python_traceback) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_multiline_method_call_b) ... ok


TracebackCases.test_recursion_error_during_traceback) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_decorator_application_lineno_correct) ... ok


TracebackErrorCaretTests.test_caret_in_type_annotation) ... ok


TracebackException.test_no_refs_to_exception_and_traceback_objects) ... ok


TracebackFormatTests.test_recursive_traceback_cpython_internal) ... ok


TracebackFormatTests.test_stack_format) ... ok


TracebackErrorCaretTests.test_wide_characters_unicode_with_problematic_byte_offset) ... ok


TracebackCases.test_caret) ... ok


TracebackCases.test_format_exception_exc) ... ok


TracebackErrorCaretTests.test_secondary_caret_not_elided) ... ok


TracebackException.test_from_exception) ... ok


TracebackCases.test_print_exception) ... ok


TracebackException.test_print) ... ok


TracebackException.test_no_locals) ... ok


TracebackCases.test_clear) ... ok


TracebackCases.test_format_exception_only_bad__str__) ... ok


TracebackCases.test_exception_is_None) ... ok


TracebackException.test_lookup_lines) ... ok


TracebackException_ExceptionGroups.test_max_group_width) ... ok


TracebackCases.test_bad_indentation) ... ok


TracebackException.test_traceback_header) ... ok


TracebackCases.test_signatures) ... ok


TracebackException_ExceptionGroups.test_exception_group_format) ... ok


TracebackFormatTests.test_print_stack) ... ok


TracebackErrorCaretTests.test_caret_for_binary_operators_with_unicode) ... ok


TracebackErrorCaretTests.test_line_with_unicode) ... ok


TracebackFormatTests.test_exception_group_deep_recursion_capi) ... ok


TracebackException.test_limit) ... ok


TracebackErrorCaretTests.test_caret_for_binary_operators) ... ok


TracebackException.test_comparison_basic) ... ok


TracebackErrorCaretTests.test_byte_offset_with_wide_characters_middle) ... ok


TracebackErrorCaretTests.test_caret_for_subscript) ... ok


TracebackException_ExceptionGroups.test_comparison) ... ok


TracebackErrorCaretTests.test_byte_offset_multiline) ... ok


TracebackException.test_smoke) ... ok


TracebackErrorCaretTests.test_caret_multiline_expression_bin_op) ... ok


TracebackCases.test_encoded_file) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_caret_for_binary_operators_two_char) ... ok


TracebackFormatTests.test_traceback_format_with_cleared_frames) ... ok


TracebackFormatTests.test_unhashable) ... ok


TracebackException_ExceptionGroups.test_max_group_depth) ... ok


TracebackException.test_cause) ... ok


TracebackFormatTests.test_traceback_format) ... ok


TracebackException.test_comparison_params_variations) ... ok


TracebackException.test_context) ... ok


TracebackException_ExceptionGroups.test_exception_group_construction) ... ok


TracebackErrorCaretTests.test_caret_multiline_expression_syntax_error) ... ok


TracebackException.test_unhashable) ... ok


TracebackCases.test_print_exception_exc) ... ok

kumaraditya303 added a commit to kumaraditya303/cpython that referenced this pull request Nov 24, 2022
@kumaraditya303
Revert "pythonGH-66285: fix forking in asyncio (python#99539)" …
0671a90 
This reverts commit 0c1fbc1.
@brettcannon

brettcannon commented Nov 24, 2022

Copy link
Copy Markdown
Member

Looks like the failure was random as subsequent builds succeeded.

https://buildbot.python.org/all/#/builders/1046

@vstinner

vstinner commented Nov 25, 2022

Copy link
Copy Markdown
Member

Yeah, I saw the test_traceback failure, but it only occurred once, so I ignored it.

@wjmelements wjmelements mentioned this pull request Feb 22, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gvanrossum gvanrossum gvanrossum approved these changes

@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner

@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner

Assignees

No one assigned

Labels

topic-asyncio

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

asyncio signal handler receives signals from child processes asyncio: support multiprocessing (support fork)

5 participants

@kumaraditya303 @bedevere-bot @gvanrossum @brettcannon @vstinner