gh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation#97644

iritkatriel

Brings the code closer to the organisation discussed on the issue (with the exception of line number calculations which is not yet safe to move).

Also a few minor changes, like renaming a couple of functions for clarity.

Issue: Split compiler into code-gen, optimizer and assembler. #87092

…as calculated

…from two places

…ps. rename calculate_jump_targets --> translate_jump_labels_to_targets

markshannon

Looks good in general, a few quibbles about naming and some comments.

I don't think "desugaring" is the correct term here. Desugaring implies that there is an equivalent source form, but there is not.
I can't think of a one word term for this set of operations, but I don't think we need one.

markshannon

Thanks

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @iritkatriel for commit e44b087 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @iritkatriel for commit 4f15173 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 Windows Non-Debug 3.x has failed when building commit c529b45.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/730/builds/5950) and take a look at the build logs.
Check if the failure is related to this commit (c529b45) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/730/builds/5950

Failed tests:

test_peg_generator

Summary of the results of the build (if available):

== Tests result: FAILURE then SUCCESS ==

403 tests OK.

10 slowest tests:

test_multiprocessing_spawn: 1 min 53 sec
test_asyncio: 1 min 39 sec
test_concurrent_futures: 1 min 15 sec
test_distutils: 1 min 15 sec
test_socket: 1 min 6 sec
test_urllib2_localnet: 44.7 sec
test_mmap: 41.2 sec
test_imaplib: 34.5 sec
test_venv: 32.2 sec
test_xmlrpc: 31.7 sec

34 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_perf_profiler test_pipes
test_poll test_posix test_pty test_pwd test_readline test_resource
test_spwd test_syslog test_threadsignals test_tix test_tkinter
test_ttk test_wait3 test_wait4 test_xxtestfuzz test_zipfile64

1 re-run test:
test_peg_generator

Total duration: 21 min 23 sec

Click to see traceback logs

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\__main__.py", line 2, in <module>
    main()
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\libregrtest\main.py", line 763, in main
    Regrtest().main(tests=tests, **kwargs)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\libregrtest\main.py", line 695, in main
    with os_helper.temp_cwd(test_cwd, quiet=True):
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(value)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 534, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(value)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 488, in temp_dir
    rmtree(path)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 445, in rmtree
    _rmtree(path)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 388, in _rmtree
    _waitfor(_rmtree_inner, path, waitall=True)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 333, in _waitfor
    func(pathname)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 384, in _rmtree_inner
    _waitfor(_rmtree_inner, fullname, waitall=True)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 333, in _waitfor
    func(pathname)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 385, in _rmtree_inner
    _force_run(fullname, os.rmdir, fullname)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\__init__.py", line 214, in _force_run
    return func(*args)
           ^^^^^^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Workspace\\buildarea\\3.x.linaro-win-arm64.nondebug\\build\\build\\test_python_10364�\\test_python_worker_3148�\\tmptian4i4t'


Traceback (most recent call last):
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\__init__.py", line 203, in _force_run
    return func(*args)
           ^^^^^^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Workspace\\buildarea\\3.x.linaro-win-arm64.nondebug\\build\\build\\test_python_10364�\\test_python_worker_3148�\\tmptian4i4t'


Traceback (most recent call last):
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 483, in temp_dir
    yield path
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\support\os_helper.py", line 536, in temp_cwd
    yield cwd_dir
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\libregrtest\main.py", line 701, in main
    self._main(tests, kwargs)
  File "C:\Workspace\buildarea\3.x.linaro-win-arm64.nondebug\build\Lib\test\libregrtest\main.py", line 758, in _main
    sys.exit(0)
SystemExit: 0

* main: (66 commits) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896) pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383) pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759) pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251) pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662) pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644) pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756) pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778) pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826) Add re.VERBOSE flag documentation example (python#97678) ...

…embler organisation (pythonGH-97644)

iritkatriel added 8 commits September 29, 2022 10:58

add unit test for const trimming

27102b2

move trim_unused_consts to assembly stage

9827030

move a few things around

103c9ae

remove redundant assert and move another up to just after the value w… …

bab900a

…as calculated

move push_cold_blocks_to_end up

9736e75

reduce repetition: create basicblock_append_instructions and call it … …

eb6cd76

…from two places

move remove_redundant_nops call into convert_exception_handlers_to_no… …

fa0b95d

…ps. rename calculate_jump_targets --> translate_jump_labels_to_targets

move check_loads function to optimization stage

8a2f28e

iritkatriel requested a review from markshannon as a code owner September 29, 2022 13:03

bedevere-bot added the awaiting core review label Sep 29, 2022

iritkatriel self-assigned this Sep 29, 2022

iritkatriel added the skip news label Sep 29, 2022

markshannon reviewed Oct 3, 2022

View reviewed changes

iritkatriel added 3 commits October 3, 2022 22:18

mark's review comments

85751ea

fix comment

5f78cc7

Call it preprocessing

e44b087

markshannon approved these changes Oct 4, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 4, 2022

iritkatriel added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 4, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 4, 2022

Merge branch 'main' into compiler_refactor

4f15173

iritkatriel added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 4, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 4, 2022

iritkatriel changed the title ~~gh-87092: bring compiler code closer to a desugaring-opt-assembler organisation~~ Oct 5, 2022

iritkatriel merged commit c529b45 into python:main Oct 5, 2022

bedevere-bot removed the awaiting merge label Oct 5, 2022

mpage pushed a commit to mpage/cpython that referenced this pull request Oct 11, 2022

pythongh-87092: bring compiler code closer to a preprocessing-opt-ass… …

6d4d702

…embler organisation (pythonGH-97644)

iritkatriel deleted the compiler_refactor branch October 18, 2022 14:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation#97644

gh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation#97644
iritkatriel merged 12 commits into
python:mainfrom
iritkatriel:compiler_refactor

iritkatriel commented Sep 29, 2022 •

edited by bedevere-bot

Loading

Uh oh!

markshannon left a comment

Uh oh!

markshannon left a comment

Uh oh!

bedevere-bot commented Oct 4, 2022

Uh oh!

bedevere-bot commented Oct 4, 2022

Uh oh!

bedevere-bot commented Oct 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

iritkatriel commented Sep 29, 2022 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Oct 4, 2022

Uh oh!

bedevere-bot commented Oct 4, 2022

Uh oh!

bedevere-bot commented Oct 5, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

iritkatriel commented Sep 29, 2022 •

edited by bedevere-bot

Loading