gh-104909: Split some more insts into ops#109943

gvanrossum

I started by adding some code that shows the most deserving non-viable opcodes. Also doubling max trace size again.

I could do this all day, but the stats show there are diminishing returns. Output from python3 Tools/cases_generator/generate_cases.py -v before splitting the top 5:

Read 186 instructions, 46 ops, 207 macros, 12 pseudos, and 14 families from Python/bytecodes.c
The following ops are not viable uops:
      1,906,418,185 LOAD_ATTR_METHOD_WITH_VALUES        (specialization of LOAD_ATTR)
      1,408,928,326 LOAD_ATTR_METHOD_NO_DICT            (specialization of LOAD_ATTR)
      1,359,753,529 LOAD_ATTR_SLOT                      (specialization of LOAD_ATTR)
      1,064,113,118 STORE_ATTR_SLOT                     (specialization of STORE_ATTR)
        981,677,187 STORE_ATTR_INSTANCE_VALUE           (specialization of STORE_ATTR)
        958,283,266 YIELD_VALUE                        
        777,829,833 CALL                                     (unspecialized)
        490,635,610 SEND_GEN                            (specialization of SEND)
        436,706,438 JUMP_FORWARD                       
        430,588,960 EXTENDED_ARG                       
        391,017,977 JUMP_BACKWARD_NO_INTERRUPT         
        351,940,012 LOAD_ATTR_MODULE                    (specialization of LOAD_ATTR)
        347,159,058 LOAD_ATTR_WITH_HINT                 (specialization of LOAD_ATTR)
        308,391,927 FOR_ITER                                 (unspecialized)
        251,204,370 RETURN_GENERATOR                   
        241,313,761 CALL_LIST_APPEND                    (specialization of CALL)
        177,791,632 CALL_PY_WITH_DEFAULTS               (specialization of CALL)
        175,204,354 CALL_KW                            
        163,262,604 FOR_ITER_GEN                        (specialization of FOR_ITER)
        146,454,299 BINARY_SUBSCR_GETITEM               (specialization of BINARY_SUBSCR)
        143,423,491 LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES (specialization of LOAD_ATTR)
        136,661,968 STORE_FAST_LOAD_FAST               
        136,511,856 LOAD_ATTR_CLASS                     (specialization of LOAD_ATTR)
        112,730,831 SEND                                     (unspecialized)
         92,715,546 MAKE_CELL                          
         83,410,408 CALL_FUNCTION_EX                   
         69,188,245 CALL_ALLOC_AND_ENTER_INIT           (specialization of CALL)
         56,944,041 LOAD_ATTR_METHOD_LAZY_DICT          (specialization of LOAD_ATTR)
         52,280,673 LOAD_ATTR_PROPERTY                  (specialization of LOAD_ATTR)
         50,964,787 STORE_ATTR_WITH_HINT                (specialization of STORE_ATTR)
         22,352,640 LOAD_ATTR_NONDESCRIPTOR_NO_DICT     (specialization of LOAD_ATTR)
          8,497,626 RERAISE                            
          6,000,000 END_ASYNC_FOR                      
          5,253,976 BEFORE_WITH                        
          2,897,006 RAISE_VARARGS                      
          2,005,368 IMPORT_FROM                        
          1,968,468 IMPORT_NAME                        
              8,160 BEFORE_ASYNC_WITH                  
              5,907 RESUME                                   (unspecialized)
              1,540 LOAD_SUPER_ATTR                          (unspecialized)
                801 CLEANUP_THROW                      
                    ENTER_EXECUTOR                     
                    LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN   (specialization of LOAD_ATTR)

I took the counts from here (follow the pystats raw link).

Curious what's up with LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN? It was the same on the weekly run a week before; the week before that (Sept 10) the file format was different. @brandtbucher any intuition on this?

Issue: Split opcodes into micro-ops #104909

gvanrossum

Another couple of questions for @brandtbucher (or @markshannon).

Sometimes there's duplication between two uops, e.g. _GUARD_DORV_VALUES and _STORE_ATTR_INSTANCE_VALUE both set dorv = *_PyObject_DictOrValuesPointer(owner);. In Tier 1 the assumption has always been that the C compiler's optimizer will eliminate the duplication (it's a static inline function). But in the Tier 2 interpreter, we end up executing that code twice (once in each uop), and in the copy-and-patch machine code it also takes up twice the space. It doesn't look too bad, because after removing asserts it's just obj - 3, and a modern CPU should do this very fast, but there might be other occurrences of this kind of duplication that are slightly less minimal.
There's an assert(type_version != 0); in the _GUARD_TYPE_VERSION macro. This existed in most of the instructions I split up, except in LOAD_ATTR_METHOD_NO_DICT. Was that an oversight, or can it be zero for that opcode? (If so, then my PR is incorrect.)
In _LOAD_ATTR_SLOT, which is an "action" uop, I left a DEOPT_IF call. This is unprincipled, but it seems to work, and splitting this off into its own guard would duplicate much of the work this uop does (in particular the indirect load). Thoughts?

Fidget-Spinner

In _LOAD_ATTR_SLOT, which is an "action" uop, I left a DEOPT_IF call. This is unprincipled, but it seems to work, and splitting this off into its own guard would duplicate much of the work this uop does (in particular the indirect load). Thoughts?

Not Mark or Brandt, but on the optimization side, we would have to treat this as an impure op, which makes most of the optimization opportunities go away. However, we're likely converting some of these to LOAD_CONST anyways so that might mitigate it?

gvanrossum

Not Mark or Brandt, but on the optimization side, we would have to treat this as an impure op, which makes most of the optimization opportunities go away.

We already have other actions using DEOPT_IF (e.g. _LOAD_GLOBAL_BUILTINS). We can split all these up later. My first goal is just to be able to translate more bytecodes to Tier 2, I'm slowly working my way through the most frequently executed instructions (per the table shown above).

However, we're likely converting some of these to LOAD_CONST anyways so that might mitigate it?

That's still a bit in the future though. And I'm not sure how often LOAD_ATTR_SLOT in particular would become LOAD_CONST (since it specifically fetches an instance attribute). It seems more likely for some other variants like LOAD_ATTR_MODULE, and that would probably load the result from the cache, not be transformed in (some guard +) LOAD_CONST.

gvanrossum

Going to merge this soon, basically as-is. Perf is same as base (1.00 faster).

gvanrossum

I can't find a persistent pattern in the failing tests. Maybe it's a timeout in multiprocessing? I believe @vstinner recently made flaky tests more likely to cause CI failure, maybe it's that?

gvanrossum

Well, okay, some tests fail consistently, so I have to bisect. Sorry for the accusations.

gvanrossum

Looks like LOAD_ATTR_METHOD_WITH_VALUES is the culprit.

I had dropped one of the DEOPT_IF() calls! :-(

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable LTO + PGO 3.x has failed when building commit 5bb6f0f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/29/builds/4838) and take a look at the build logs.
Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/29/builds/4838

Failed tests:

test.test_concurrent_futures.test_shutdown

Failed subtests:

test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError: b'Exception in thread Thread-1:\nTraceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/threading.py", line 1066, in _bootstrap_inner\n    self.run()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/concurrent/futures/process.py", line 344, in run\n    self.add_call_item_to_queue()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/concurrent/futures/process.py", line 399, in add_call_item_to_queue\n    self.call_queue.put(_CallItem(work_id,\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/queues.py", line 94, in put\n    self._start_thread()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/queues.py", line 177, in _start_thread\n    self._thread.start()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/threading.py", line 985, in start\n    _start_new_thread(self._bootstrap, ())\nRuntimeError: can\'t create new thread at interpreter shutdown\nTraceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 274, in main\n    code = _serve_one(child_r, fds,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one\n    code = spawn._main(child_r, parent_sentinel)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/spawn.py", line 132, in _main\n    self = reduction.pickle.load(from_parent)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__\n    self._semlock = _multiprocessing.SemLock._rebuild(*state)\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n' is not false

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian PGO 3.x has failed when building commit 5bb6f0f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/249/builds/6464) and take a look at the build logs.
Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/249/builds/6464

Failed tests:

test.test_multiprocessing_fork.test_processes

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 29, done.        
remote: Counting objects:   3% (1/29)        
remote: Counting objects:   6% (2/29)        
remote: Counting objects:  10% (3/29)        
remote: Counting objects:  13% (4/29)        
remote: Counting objects:  17% (5/29)        
remote: Counting objects:  20% (6/29)        
remote: Counting objects:  24% (7/29)        
remote: Counting objects:  27% (8/29)        
remote: Counting objects:  31% (9/29)        
remote: Counting objects:  34% (10/29)        
remote: Counting objects:  37% (11/29)        
remote: Counting objects:  41% (12/29)        
remote: Counting objects:  44% (13/29)        
remote: Counting objects:  48% (14/29)        
remote: Counting objects:  51% (15/29)        
remote: Counting objects:  55% (16/29)        
remote: Counting objects:  58% (17/29)        
remote: Counting objects:  62% (18/29)        
remote: Counting objects:  65% (19/29)        
remote: Counting objects:  68% (20/29)        
remote: Counting objects:  72% (21/29)        
remote: Counting objects:  75% (22/29)        
remote: Counting objects:  79% (23/29)        
remote: Counting objects:  82% (24/29)        
remote: Counting objects:  86% (25/29)        
remote: Counting objects:  89% (26/29)        
remote: Counting objects:  93% (27/29)        
remote: Counting objects:  96% (28/29)        
remote: Counting objects: 100% (29/29)        
remote: Counting objects: 100% (29/29), done.        
remote: Compressing objects:   6% (1/15)        
remote: Compressing objects:  13% (2/15)        
remote: Compressing objects:  20% (3/15)        
remote: Compressing objects:  26% (4/15)        
remote: Compressing objects:  33% (5/15)        
remote: Compressing objects:  40% (6/15)        
remote: Compressing objects:  46% (7/15)        
remote: Compressing objects:  53% (8/15)        
remote: Compressing objects:  60% (9/15)        
remote: Compressing objects:  66% (10/15)        
remote: Compressing objects:  73% (11/15)        
remote: Compressing objects:  80% (12/15)        
remote: Compressing objects:  86% (13/15)        
remote: Compressing objects:  93% (14/15)        
remote: Compressing objects: 100% (15/15)        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 15 (delta 14), reused 1 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '5bb6f0fcba663e1006f9063d1027ce8bd9f8effb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 5bb6f0fcba gh-104909: Split some more insts into ops (#109943)
Switched to and reset branch 'main'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:2821: clean-retain-profile] Error 1 (ignored)
In file included from Python/ceval.c:766:
Python/generated_cases.c.h: In function ‘_PyEval_EvalFrameDefault’:
Python/generated_cases.c.h:2634:31: warning: unused variable ‘tp’ [-Wunused-variable]
 2634 |                 PyTypeObject *tp = Py_TYPE(owner);
      |                               ^~
Python/generated_cases.c.h:3593:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3593 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
Python/generated_cases.c.h:3642:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3642 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
In file included from Python/executor.c:87:
Python/executor_cases.c.h: In function ‘_PyUopExecute’:
Python/executor_cases.c.h:1739:27: warning: unused variable ‘tp’ [-Wunused-variable]
 1739 |             PyTypeObject *tp = Py_TYPE(owner);
      |                           ^~
Python/executor_cases.c.h:2302:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2302 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
Python/executor_cases.c.h:2348:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2348 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
./Modules/readline.c: In function ‘setup_readline’:
./Modules/readline.c:1256:21: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1256 |     rl_startup_hook = on_startup_hook;
      |                     ^
./Modules/readline.c:1258:23: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1258 |     rl_pre_input_hook = on_pre_input_hook;
      |                       ^
./Modules/socketmodule.c: In function ‘getsockaddrarg’:
./Modules/socketmodule.c:2468:9: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 2468 |         strncpy((char *)sa->salg_name, name, sizeof(sa->salg_name));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/_decimal/libmpdec/context.c:56: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time

./Modules/_decimal/libmpdec/context.c:56: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time

In file included from Python/ceval.c:766:
Python/generated_cases.c.h: In function ‘_PyEval_EvalFrameDefault’:
Python/generated_cases.c.h:2634:31: warning: unused variable ‘tp’ [-Wunused-variable]
 2634 |                 PyTypeObject *tp = Py_TYPE(owner);
      |                               ^~
Python/generated_cases.c.h:3593:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3593 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
Python/generated_cases.c.h:3642:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3642 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
In file included from Python/executor.c:87:
Python/executor_cases.c.h: In function ‘_PyUopExecute’:
Python/executor_cases.c.h:1739:27: warning: unused variable ‘tp’ [-Wunused-variable]
 1739 |             PyTypeObject *tp = Py_TYPE(owner);
      |                           ^~
Python/executor_cases.c.h:2302:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2302 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
Python/executor_cases.c.h:2348:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2348 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
./Modules/readline.c: In function ‘setup_readline’:
./Modules/readline.c:1256:21: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1256 |     rl_startup_hook = on_startup_hook;
      |                     ^
./Modules/readline.c:1258:23: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1258 |     rl_pre_input_hook = on_pre_input_hook;
      |                       ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘doParseXmlDecl’ at ./Modules/expat/xmltok.c:1197:13:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1075:9:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1081:12:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1088:9:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1115:7:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1128:9:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

make: *** [Makefile:2040: buildbottest] Error 5

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO 3.x has failed when building commit 5bb6f0f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/567/builds/4991) and take a look at the build logs.
Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/567/builds/4991

Failed tests:

test.test_multiprocessing_fork.test_processes

Summary of the results of the build (if available):

==

Click to see traceback logs

Note: switching to '5bb6f0fcba663e1006f9063d1027ce8bd9f8effb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 5bb6f0fcba gh-104909: Split some more insts into ops (#109943)
Switched to and reset branch 'main'

In file included from Python/ceval.c:766:
Python/generated_cases.c.h: In function ‘_PyEval_EvalFrameDefault’:
Python/generated_cases.c.h:2634:31: warning: unused variable ‘tp’ [-Wunused-variable]
                 PyTypeObject *tp = Py_TYPE(owner);
                               ^~
Python/generated_cases.c.h:3593:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
                 PyTypeObject *owner_cls = Py_TYPE(owner);
                               ^~~~~~~~~
Python/generated_cases.c.h:3642:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
                 PyTypeObject *owner_cls = Py_TYPE(owner);
                               ^~~~~~~~~
In file included from Python/executor.c:87:
Python/executor_cases.c.h: In function ‘_PyUopExecute’:
Python/executor_cases.c.h:1739:27: warning: unused variable ‘tp’ [-Wunused-variable]
             PyTypeObject *tp = Py_TYPE(owner);
                           ^~
Python/executor_cases.c.h:2302:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
             PyTypeObject *owner_cls = Py_TYPE(owner);
                           ^~~~~~~~~
Python/executor_cases.c.h:2348:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
             PyTypeObject *owner_cls = Py_TYPE(owner);
                           ^~~~~~~~~
./Include/internal/pycore_semaphore.h:54:1: warning: type of ‘_PySemaphore_Wakeup’ does not match original declaration [-Wlto-type-mismatch]
 _PySemaphore_Wakeup(_PySemaphore *sema);
 ^
Python/parking_lot.c:197:1: note: ‘_PySemaphore_Wakeup’ was previously declared here
 _PySemaphore_Wakeup(_PySemaphore *sema)
 ^
Python/parking_lot.c:197:1: note: code may be misoptimized unless -fno-strict-aliasing is used
./Include/internal/pycore_semaphore.h:57:18: warning: type of ‘_PySemaphore_Init’ does not match original declaration [-Wlto-type-mismatch]
 PyAPI_FUNC(void) _PySemaphore_Init(_PySemaphore *sema);
                  ^
Python/parking_lot.c:53:1: note: ‘_PySemaphore_Init’ was previously declared here
 _PySemaphore_Init(_PySemaphore *sema)
 ^
Python/parking_lot.c:53:1: note: code may be misoptimized unless -fno-strict-aliasing is used
./Include/internal/pycore_semaphore.h:49:1: warning: type of ‘_PySemaphore_Wait’ does not match original declaration [-Wlto-type-mismatch]
 _PySemaphore_Wait(_PySemaphore *sema, _PyTime_t timeout_ns, int detach);
 ^
Python/parking_lot.c:178:1: note: ‘_PySemaphore_Wait’ was previously declared here
 _PySemaphore_Wait(_PySemaphore *sema, _PyTime_t timeout, int detach)
 ^
Python/parking_lot.c:178:1: note: code may be misoptimized unless -fno-strict-aliasing is used
./Include/internal/pycore_semaphore.h:58:18: warning: type of ‘_PySemaphore_Destroy’ does not match original declaration [-Wlto-type-mismatch]
 PyAPI_FUNC(void) _PySemaphore_Destroy(_PySemaphore *sema);
                  ^
Python/parking_lot.c:81:1: note: ‘_PySemaphore_Destroy’ was previously declared here
 _PySemaphore_Destroy(_PySemaphore *sema)
 ^
Python/parking_lot.c:81:1: note: code may be misoptimized unless -fno-strict-aliasing is used

make: *** [Makefile:2040: buildbottest] Error 5

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian 3.x has failed when building commit 5bb6f0f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/424/builds/5033) and take a look at the build logs.
Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/424/builds/5033

Failed tests:

test.test_multiprocessing_spawn.test_misc

Failed subtests:

test_dict - test.test_multiprocessing_spawn.test_misc.TestSyncManagerTypes.test_dict
test_bounded_semaphore - test.test_multiprocessing_spawn.test_misc.TestSyncManagerTypes.test_bounded_semaphore
test_condition - test.test_multiprocessing_spawn.test_misc.TestSyncManagerTypes.test_condition

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5931, in test_dict
    self.run_worker(self._test_dict, o)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5799, in run_worker
    self.wait_proc_exit()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5781, in wait_proc_exit
    join_process(self.proc)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 100, in join_process
    threading_helper.join_thread(process)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/threading_helper.py", line 113, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 52.5 seconds


Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5854, in test_condition
    self.run_worker(self._test_condition, o)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5799, in run_worker
    self.wait_proc_exit()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5781, in wait_proc_exit
    join_process(self.proc)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 100, in join_process
    threading_helper.join_thread(process)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/threading_helper.py", line 113, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 52.5 seconds


Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5845, in test_bounded_semaphore
    self.test_semaphore(sname="BoundedSemaphore")
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5841, in test_semaphore
    self.run_worker(self._test_semaphore, o)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5799, in run_worker
    self.wait_proc_exit()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5781, in wait_proc_exit
    join_process(self.proc)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 100, in join_process
    threading_helper.join_thread(process)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/threading_helper.py", line 113, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 52.5 seconds

These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(

vstinner

I believe @vstinner recently made flaky tests more likely to cause CI failure, maybe it's that?

Right. Tests are now run with --rerun-fail on GitHub Actions and buildbots: any test marks the whole build as a failure.

These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR` that weren't already viable uops: * Split LOAD_ATTR_METHOD_WITH_VALUES * Split LOAD_ATTR_METHOD_NO_DICT * Split LOAD_ATTR_SLOT * Split STORE_ATTR_SLOT * Split STORE_ATTR_INSTANCE_VALUE Also: * Add `-v` flag to code generator which prints a list of non-viable uops (easter-egg: it can print execution counts -- see source) * Double _Py_UOP_MAX_TRACE_LENGTH to 128 I had dropped one of the DEOPT_IF() calls! :-(

gvanrossum added 8 commits September 26, 2023 17:43

Print non-viable uops

2ca26bb

Sort non-viable opcodes by execution count

e19517c

Double _Py_UOP_MAX_TRACE_LENGTH to 128

8d1a856

Split LOAD_ATTR_METHOD_WITH_VALUES

3796e89

Split LOAD_ATTR_METHOD_NO_DICT

6d816ed

Split LOAD_ATTR_SLOT (weird: action may deopt)

5ba03d1

Split STORE_ATTR_SLOT

5d0fa92

Split STORE_ATTR_INSTANCE_VALUE

b54eee3

gvanrossum force-pushed the load-attr-uops branch from 7bdddb0 to b54eee3 Compare September 27, 2023 04:27

gvanrossum added the skip news label Sep 27, 2023

gvanrossum changed the title ~~Split some more insts into ops~~ Sep 27, 2023

bedevere-app Bot mentioned this pull request Sep 27, 2023

Split opcodes into micro-ops #104909

Closed

markshannon reviewed Sep 27, 2023

View reviewed changes

gvanrossum marked this pull request as ready for review September 27, 2023 17:44

bedevere-app Bot added the awaiting review label Sep 27, 2023

Merge branch 'main' into load-attr-uops

24b4ec9

Fix LOAD_ATTR_METHOD_WITH_VALUES …

85fb1e7

I had dropped one of the DEOPT_IF() calls! :-(

gvanrossum merged commit 5bb6f0f into python:main Sep 27, 2023

bedevere-app Bot removed the awaiting review label Sep 27, 2023

gvanrossum deleted the load-attr-uops branch September 27, 2023 22:27

gvanrossum mentioned this pull request Sep 28, 2023

How to handle YIELD_VALUE in Tier 2 faster-cpython/ideas#628

Open

This was referenced Sep 28, 2023

Warning: unused variable ‘owner_cls’ in generated_cases.c.h #110020

Closed

gh-110020: Fix unused variable warnings in bytecodes.c #110023

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-104909: Split some more insts into ops#109943