gh-109566, regrtest: Add --fast-ci and --slow-ci options#109570

vstinner

Add --fast-ci and --slow-ci options to libregrtest:
- --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
  (skip slowest tests).
- --slow-ci uses a default timeout of 20 minues and "-u all" (run
  all tests).
regrtest header now lists test resources.
Makefile changes:
- "make test", "make hostrunnertest" and "make coverage-report" now
  use --fast-ci option and TESTTIMEOUT variable.
- "make buildbottest" now uses "--slow-ci". Remove options which
  became redundant with "--slow-ci".
- "make testall" and "make testuniversal" now use --slow-ci option
  and TESTTIMEOUT variable.
- "make testall" now uses "find -exec rm ..." instead of
  "find ... -print|xargs rm ...", same as "make clean".
GitHub Actions workflow:
- Ubuntu and Address Sanitizer jobs now use "make test". Remove
  options which became redundant with "--fast-ci".
- Windows jobs now use --fast-ci option.
- Use -j0 to detect the number of CPUs.
Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
Tools/scripts/run_tests.py now uses --fast-ci option.
Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.

Issue: Unify ways to run the Python test suite #109566

📚 Documentation preview 📚: https://cpython-previews--109570.org.readthedocs.build/

vstinner

I'm not sure that these 2 options are required. Maybe --fast-ci should simply be the default in regrtest.

I'm not sure that we need some many ways to run tests. Why not having better default behavior, without any option?

cc @serhiy-storchaka @gpshead @zware @pablogsal

vstinner

When Python is build to create a Linux Fedora package, we run the test suite with ./python -m test. In this case, we prefer to tolerate unstable tests. The packager role is not to fix the test failures, but just package Python.

Maybe that's where we can put a line between "strict" and "not strict":

Strict: don't tolerate any test failure, use --fail-env-changed and --fail-rerun.
Not strict: tolerate tests failing randomly, use --rerun and treat "fail than pass" as SUCCESS.

vstinner

--fast-ci is designed for the Python workflow (GitHub Actions workflow): quick tests to validate a pull request.
--slow-ci is designed for Continuous Integration running in the background (buildbots), not blocked pull requests, where we can allow tests to run (way) longer.

For example, in time budget, I would be fine with --fast-ci taking up to 15 min, whereas I don't bother if --slow-ci takes 30 min to 2 hours.

Obviously, by design, some tests can pass with --fast-ci, and fail with --slow-ci. That's how I managed to recently merge a pull request breaking test_peg_generator which an obvious (and stupid) bug (in my large regrtest refactoring work).

serhiy-storchaka

So, --fast-ci is equivalent to -uall,-cpu --timeout=60, and --slow-ci is equivalent to -uall --timeout=120? What is the benefit? Why not simply use existing options -u and --timeout?

vstinner

So, --fast-ci is equivalent to -uall,-cpu --timeout=60, and --slow-ci is equivalent to -uall --timeout=120? What is the benefit?

Yes, but they also add more options:

-j0 --rerun -r --fail-env-changed --fail-rerun --slowest --verbose3 --nowindows

Later, I also plan to attempt to pass options to Python like:

-u -Wd -E -bb

But this change is more complicated because of WASM/WASI buildbot workers which use a "host runner" thing and --python option. I prefer to do it in a separated PR.

vstinner

--fail-rerun did its job on Windows x86: test_concurrent_futures.test_as_completed.test_future_times_out() failed, see issue #109565.

* Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.

vstinner

I removed make fastcitest and instead modified make test to use --fast-ci option. There are already too many ways to run tests, I prefer to avoid adding a new way.

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 4fcd6af 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

vstinner

ARM64 Windows PR

Git step failed.

s390x Fedora Clang PR
s390x RHEL8 PR

test_tools failed: issue gh-109615.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD 3.x has failed when building commit 859618c.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1223/builds/230) and take a look at the build logs.
Check if the failure is related to this commit (859618c) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1223/builds/230

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/threading.py", line 1066, in _bootstrap_inner
    self.run()
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/threading.py", line 1003, in run
    self._target(*self._args, **self._kwargs)
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/test_interpreters.py", line 483, in task
    interp = interpreters.create()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/support/interpreters.py", line 25, in create
    id = _interpreters.create(isolated=isolated)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: interpreter creation failed
k


Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1316, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1256, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1538, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1512, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1630, in find_spec
  File "<frozen importlib._bootstrap_external>", line 161, in _path_isfile
  File "<frozen importlib._bootstrap_external>", line 153, in _path_is_mode_type
  File "<frozen importlib._bootstrap_external>", line 147, in _path_stat
ValueError: embedded null byte

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 macOS 3.x has failed when building commit 859618c.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/725/builds/5746) and take a look at the build logs.
Check if the failure is related to this commit (859618c) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/725/builds/5746

Failed tests:

test.test_concurrent_futures.test_deadlock

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 52, done.        
remote: Counting objects:   1% (1/52)        
remote: Counting objects:   3% (2/52)        
remote: Counting objects:   5% (3/52)        
remote: Counting objects:   7% (4/52)        
remote: Counting objects:   9% (5/52)        
remote: Counting objects:  11% (6/52)        
remote: Counting objects:  13% (7/52)        
remote: Counting objects:  15% (8/52)        
remote: Counting objects:  17% (9/52)        
remote: Counting objects:  19% (10/52)        
remote: Counting objects:  21% (11/52)        
remote: Counting objects:  23% (12/52)        
remote: Counting objects:  25% (13/52)        
remote: Counting objects:  26% (14/52)        
remote: Counting objects:  28% (15/52)        
remote: Counting objects:  30% (16/52)        
remote: Counting objects:  32% (17/52)        
remote: Counting objects:  34% (18/52)        
remote: Counting objects:  36% (19/52)        
remote: Counting objects:  38% (20/52)        
remote: Counting objects:  40% (21/52)        
remote: Counting objects:  42% (22/52)        
remote: Counting objects:  44% (23/52)        
remote: Counting objects:  46% (24/52)        
remote: Counting objects:  48% (25/52)        
remote: Counting objects:  50% (26/52)        
remote: Counting objects:  51% (27/52)        
remote: Counting objects:  53% (28/52)        
remote: Counting objects:  55% (29/52)        
remote: Counting objects:  57% (30/52)        
remote: Counting objects:  59% (31/52)        
remote: Counting objects:  61% (32/52)        
remote: Counting objects:  63% (33/52)        
remote: Counting objects:  65% (34/52)        
remote: Counting objects:  67% (35/52)        
remote: Counting objects:  69% (36/52)        
remote: Counting objects:  71% (37/52)        
remote: Counting objects:  73% (38/52)        
remote: Counting objects:  75% (39/52)        
remote: Counting objects:  76% (40/52)        
remote: Counting objects:  78% (41/52)        
remote: Counting objects:  80% (42/52)        
remote: Counting objects:  82% (43/52)        
remote: Counting objects:  84% (44/52)        
remote: Counting objects:  86% (45/52)        
remote: Counting objects:  88% (46/52)        
remote: Counting objects:  90% (47/52)        
remote: Counting objects:  92% (48/52)        
remote: Counting objects:  94% (49/52)        
remote: Counting objects:  96% (50/52)        
remote: Counting objects:  98% (51/52)        
remote: Counting objects: 100% (52/52)        
remote: Counting objects: 100% (52/52), done.        
remote: Compressing objects:   4% (1/24)        
remote: Compressing objects:   8% (2/24)        
remote: Compressing objects:  12% (3/24)        
remote: Compressing objects:  16% (4/24)        
remote: Compressing objects:  20% (5/24)        
remote: Compressing objects:  25% (6/24)        
remote: Compressing objects:  29% (7/24)        
remote: Compressing objects:  33% (8/24)        
remote: Compressing objects:  37% (9/24)        
remote: Compressing objects:  41% (10/24)        
remote: Compressing objects:  45% (11/24)        
remote: Compressing objects:  50% (12/24)        
remote: Compressing objects:  54% (13/24)        
remote: Compressing objects:  58% (14/24)        
remote: Compressing objects:  62% (15/24)        
remote: Compressing objects:  66% (16/24)        
remote: Compressing objects:  70% (17/24)        
remote: Compressing objects:  75% (18/24)        
remote: Compressing objects:  79% (19/24)        
remote: Compressing objects:  83% (20/24)        
remote: Compressing objects:  87% (21/24)        
remote: Compressing objects:  91% (22/24)        
remote: Compressing objects:  95% (23/24)        
remote: Compressing objects: 100% (24/24)        
remote: Compressing objects: 100% (24/24), done.        
remote: Total 27 (delta 25), reused 4 (delta 3), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '859618c8cd5de86a975e68d7e5d20c04bc5db2e5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 859618c8cd gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
Switched to and reset branch 'main'

./Modules/readline.c:1256:21: warning: incompatible function pointer types assigning to 'Function *' (aka 'int (*)(const char *, int)') from 'int (void)' [-Wincompatible-function-pointer-types]
    rl_startup_hook = on_startup_hook;
                    ^ ~~~~~~~~~~~~~~~
./Modules/readline.c:1258:23: warning: incompatible function pointer types assigning to 'Function *' (aka 'int (*)(const char *, int)') from 'int (void)' [-Wincompatible-function-pointer-types]
    rl_pre_input_hook = on_pre_input_hook;
                      ^ ~~~~~~~~~~~~~~~~~
2 warnings generated.
./Modules/_cursesmodule.c:1336:9: warning: 'is_pad' is only available on macOS 14.0 or newer [-Wunguarded-availability-new]
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ncurses.h:964:29: note: 'is_pad' has been marked as being introduced in macOS 14.0 here, but the deployment target is macOS 13.6.0
xtern NCURSES_EXPORT(bool) is_pad (const WINDOW *);            /* generated */
                            ^
./Modules/_cursesmodule.c:1336:9: note: enclose 'is_pad' in a __builtin_available check to silence this warning
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
./Modules/_cursesmodule.c:2024:9: warning: 'is_pad' is only available on macOS 14.0 or newer [-Wunguarded-availability-new]
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ncurses.h:964:29: note: 'is_pad' has been marked as being introduced in macOS 14.0 here, but the deployment target is macOS 13.6.0
xtern NCURSES_EXPORT(bool) is_pad (const WINDOW *);            /* generated */
                            ^
./Modules/_cursesmodule.c:2024:9: note: enclose 'is_pad' in a __builtin_available check to silence this warning
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
./Modules/_cursesmodule.c:2248:9: warning: 'is_pad' is only available on macOS 14.0 or newer [-Wunguarded-availability-new]
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ncurses.h:964:29: note: 'is_pad' has been marked as being introduced in macOS 14.0 here, but the deployment target is macOS 13.6.0
xtern NCURSES_EXPORT(bool) is_pad (const WINDOW *);            /* generated */
                            ^
./Modules/_cursesmodule.c:2248:9: note: enclose 'is_pad' in a __builtin_available check to silence this warning
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
./Modules/_cursesmodule.c:2324:9: warning: 'is_pad' is only available on macOS 14.0 or newer [-Wunguarded-availability-new]
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ncurses.h:964:29: note: 'is_pad' has been marked as being introduced in macOS 14.0 here, but the deployment target is macOS 13.6.0
xtern NCURSES_EXPORT(bool) is_pad (const WINDOW *);            /* generated */
                            ^
./Modules/_cursesmodule.c:2324:9: note: enclose 'is_pad' in a __builtin_available check to silence this warning
    if (py_is_pad(self->win)) {
        ^~~~~~~~~~~~~~~~~~~~
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^~~~~~
4 warnings generated.
In file included from ./Modules/_tkinter.c:52:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
In file included from ./Modules/tkappinit.c:17:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_4/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
9 warnings generated.
9 warnings generated.

make: *** [buildbottest] Error 5

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable 3.x has failed when building commit 859618c.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/125/builds/4543) and take a look at the build logs.
Check if the failure is related to this commit (859618c) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/125/builds/4543

Failed tests:

test.test_multiprocessing_forkserver.test_processes

Failed subtests:

test_rapid_restart - test.test_multiprocessing_forkserver.test_processes.WithProcessesTestManagerRestart.test_rapid_restart

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 594, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 159, in __init__
    self.listener = Listener(address=address, backlog=16)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 464, in __init__
    self._listener = SocketListener(address, family, backlog)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 607, in __init__
    self._socket.bind(address)
OSError: [Errno 98] Address already in use


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/_test_multiprocessing.py", line 3140, in test_rapid_restart
    manager.start()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 569, in start
    self._address = reader.recv()
                    ^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL7 3.x has failed when building commit 859618c.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/179/builds/5562) and take a look at the build logs.
Check if the failure is related to this commit (859618c) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/179/builds/5562

Failed tests:

test.test_asyncio.test_tasks

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_asyncio/test_tasks.py", line 3310, in test_run_coroutine_threadsafe_with_timeout
    with self.assertRaises(asyncio.TimeoutError):
AssertionError: TimeoutError not raised

vstinner

Later, I also plan to attempt to pass options to Python like: -u -Wd -E -bb

Done in follow-up change: PR gh-109909.

vstinner

Thanks for looking into the change @gpshead :-)

…n#109570) * Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.

kulikjak

This commit changed how some tests without the internet behave (in our build infrastructure, tests don't have access to the internet). For example, test that does need it is test_codecmaps_jp; it previously ended with:

test_mapping_supplemental (test.test_codecmaps_jp.TestSJISX0213Map.test_mapping_supplemental) ... 	fetching http://www.pythontest.net/unicode/SHIFT_JISX0213.TXT ...
skipped 'Could not retrieve http://www.pythontest.net/unicode/SHIFT_JISX0213.TXT'

but after this change (bisecting led me here), it ends much more ominously with:

0:10:00 load avg: 0.57 [1/1/1] test_codecmaps_jp process crashed (Exit code 1)
Timeout (0:10:00)!
Thread 0x0000000000000001 (most recent call first):
  File "/.../cpython/Lib/socket.py", line 837 in create_connection
  File "/.../cpython/Lib/http/client.py", line 984 in connect
  File "/.../cpython/Lib/http/client.py", line 1018 in send
  File "/.../cpython/Lib/http/client.py", line 1074 in _send_output
  File "/.../cpython/Lib/http/client.py", line 1314 in endheaders
  File "/.../cpython/Lib/http/client.py", line 1365 in _send_request
  File "/.../cpython/Lib/http/client.py", line 1319 in request
  File "/.../cpython/Lib/urllib/request.py", line 1318 in do_open
  File "/.../cpython/Lib/urllib/request.py", line 1347 in http_open
  File "/.../cpython/Lib/urllib/request.py", line 466 in _call_chain
  File "/.../cpython/Lib/urllib/request.py", line 506 in _open
  File "/.../cpython/Lib/urllib/request.py", line 489 in open
  File "/.../cpython/Lib/test/support/__init__.py", line 678 in open_urlresource
  File "/.../cpython/Lib/test/multibytecodec_support.py", line 299 in open_mapping_file
  File "/.../cpython/Lib/test/multibytecodec_support.py", line 294 in setUp
  File "/.../cpython/Lib/unittest/case.py", line 586 in _callSetUp
  File "/.../cpython/Lib/unittest/case.py", line 632 in run
  File "/.../cpython/Lib/unittest/case.py", line 692 in __call__
  File "/.../cpython/Lib/unittest/suite.py", line 122 in run
  File "/.../cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/.../cpython/Lib/unittest/suite.py", line 122 in run
  File "/.../cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/.../cpython/Lib/unittest/suite.py", line 122 in run
  File "/.../cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/.../cpython/Lib/unittest/runner.py", line 240 in run
  File "/.../cpython/Lib/test/support/__init__.py", line 1152 in _run_suite
  File "/.../cpython/Lib/test/support/__init__.py", line 1279 in run_unittest
  File "/.../cpython/Lib/test/libregrtest/single.py", line 36 in run_unittest
  File "/.../cpython/Lib/test/libregrtest/single.py", line 92 in test_func
  File "/.../cpython/Lib/test/libregrtest/single.py", line 48 in regrtest_runner
  File "/.../cpython/Lib/test/libregrtest/single.py", line 95 in _load_run_test
  File "/.../cpython/Lib/test/libregrtest/single.py", line 138 in _runtest_env_changed_exc
  File "/.../cpython/Lib/test/libregrtest/single.py", line 223 in _runtest
  File "/.../cpython/Lib/test/libregrtest/single.py", line 266 in run_single_test
  File "/.../cpython/Lib/test/libregrtest/single.py", line 223 in _runtest
  File "/.../cpython/Lib/test/libregrtest/single.py", line 266 in run_single_test
  File "/.../cpython/Lib/test/libregrtest/worker.py", line 82 in worker_process
  File "/.../cpython/Lib/test/libregrtest/worker.py", line 105 in main
  File "/.../cpython/Lib/test/libregrtest/worker.py", line 109 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

== Tests result: FAILURE ==

It's not a huge deal, but probably not what was intended?

kulikjak

Ah, it's probably the newly introduced default 10 minute timeout. No other codecmaps test fails like this because they all finish (are skipped) before 10 minutes, and when I add --timeout=1200, it works again.

Sorry for the noise.

vstinner

You can add -u-network option to disable the network test resource which overrides -uall implied by slow/fast CI option. Moreover, at startup, regrtest now lists enabled test resources.

kulikjak

We are using -network and it's listed in resources:

== CPython 3.13.0a1+ (main, Nov 7 2023, 17:55:11) [GCC 13.2.0]
== Solaris-2.11-sun4v-sparc-64bit big-endian
== Python build: release shared dtrace
== cwd: /..../pythonmain/build/sparcv9/build/test_python_worker_57503æ
== CPU count: 512
== encodings: locale=UTF-8 FS=utf-8
== resources: all,-network
...

It seems that codesmaps tests are not checking whether network resource is available?

vstinner

We are using -network and it's listed in resources

all,-network means that network is negative: it's disabled.

It seems that codesmaps tests are not checking whether network resource is available?

I checked. support.open_urlresource() requires the urlfetch resource, not the network resource. Sorry. I'm not sure why there are two different resources.

gpshead

the urlfetch resource, not the network resource. Sorry. I'm not sure why there are two different resources.

I think network is the low level arbitrary socket operations one. url fetching may work in some environments (proxies?) even without arbitrary sockets. I don't know how well we actually categorize our tests that way though.

vstinner

@gpshead:

I think network is the low level arbitrary socket operations one. url fetching may work in some environments (proxies?) even without arbitrary sockets.

Oh ok, that makes sense.

kulikjak

all,-network means that network is negative: it's disabled.

I know, that is what I wanted and why I was surprised that test_codecmaps_jp is still trying to use the network.

requires the urlfetch resource, not the network resource

Indeed, Thanks! With urlfetch resource disabled, the test is now skipped gracefully and much faster than previously.

…n#109570) * Add --fast-ci and --slow-ci options to libregrtest: * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu" (skip slowest tests). * --slow-ci uses a default timeout of 20 minues and "-u all" (run all tests). * regrtest header now lists test resources. * Makefile changes: * "make test", "make hostrunnertest" and "make coverage-report" now use --fast-ci option and TESTTIMEOUT variable. * "make buildbottest" now uses "--slow-ci". Remove options which became redundant with "--slow-ci". * "make testall" and "make testuniversal" now use --slow-ci option and TESTTIMEOUT variable. * "make testall" now uses "find -exec rm ..." instead of "find ... -print|xargs rm ...", same as "make clean". * GitHub Actions workflow: * Ubuntu and Address Sanitizer jobs now use "make test". Remove options which became redundant with "--fast-ci". * Windows jobs now use --fast-ci option. * Use -j0 to detect the number of CPUs. * Set Makefile TESTTIMEOUT default to an empty string, since --slow-ci and --fast-ci use different default timeout. It's now accepted to pass "--timeout=" to regrtest: treated as not timeout. * Tools/scripts/run_tests.py now uses --fast-ci option. * Tools/buildbot/test.bat now uses --slow-ci option. Remove --timeout=1200 option, redundant with --slow-ci.

bedevere-app Bot mentioned this pull request Sep 19, 2023

Unify ways to run the Python test suite #109566

Closed

vstinner mentioned this pull request Sep 24, 2023

gh-82054: Implements test.regrtest unittest sharding to pull in long tails #99637

Closed

vstinner force-pushed the regrtest_ci branch from dbbdc26 to 70d1c17 Compare September 26, 2023 10:49

vstinner marked this pull request as ready for review September 26, 2023 10:49

vstinner requested review from corona10, erlend-aasland, ezio-melotti and hugovk as code owners September 26, 2023 10:49

bedevere-app Bot added the awaiting review label Sep 26, 2023

vstinner force-pushed the regrtest_ci branch 3 times, most recently from 699a0dd to cf59c92 Compare September 26, 2023 11:46

vstinner force-pushed the regrtest_ci branch from cf59c92 to be4c7be Compare September 26, 2023 11:49

Add test_fast_ci_resource()

4fcd6af

vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 26, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 26, 2023

vstinner merged commit 859618c into python:main Sep 26, 2023

vstinner deleted the regrtest_ci branch September 26, 2023 15:22

bedevere-app Bot removed the awaiting review label Sep 26, 2023

gpshead reviewed Sep 26, 2023

View reviewed changes

asottile reviewed Sep 30, 2023

View reviewed changes

asottile mentioned this pull request Sep 30, 2023

if SOURCE_DATE_EPOCH is set, don't randomize the testsuite #110164

Closed

Conversation

vstinner commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Sep 19, 2023

Uh oh!

vstinner commented Sep 19, 2023

Uh oh!

vstinner commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented Sep 19, 2023

Uh oh!

vstinner commented Sep 19, 2023

Uh oh!

vstinner commented Sep 19, 2023

Uh oh!

vstinner commented Sep 26, 2023

Uh oh!

bedevere-bot commented Sep 26, 2023

Uh oh!

vstinner commented Sep 26, 2023

Uh oh!

bedevere-bot commented Sep 26, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 26, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 26, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 26, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

vstinner commented Sep 26, 2023

Uh oh!

vstinner commented Sep 26, 2023

Uh oh!

kulikjak commented Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kulikjak commented Nov 7, 2023

Uh oh!

vstinner commented Nov 7, 2023

Uh oh!

kulikjak commented Nov 7, 2023

Uh oh!

vstinner commented Nov 7, 2023

Uh oh!

gpshead commented Nov 7, 2023

Uh oh!

vstinner commented Nov 7, 2023

Uh oh!

kulikjak commented Nov 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

vstinner commented Sep 19, 2023 •

edited

Loading

vstinner commented Sep 19, 2023 •

edited

Loading

kulikjak commented Nov 7, 2023 •

edited

Loading