bpo-41282: (PEP 632) Load install schemes from sysconfig#24549
Conversation
|
The test failure |
Sorry, something went wrong.
|
Someday I will get around to making a PR with the diff to PyPy's version of |
Sorry, something went wrong.
|
I'm ambivalent on the deviations this makes from the plan, and while I've not thoroughly reviewed the code, I couldn't see anything worth flagging during a cursory look through the patch. |
Sorry, something went wrong.
1d485c9 to
7876985
Compare
February 28, 2021 11:22
|
Rebased to master to restart CI. |
Sorry, something went wrong.
|
This PR is stale because it has been open for 30 days with no activity. |
Sorry, something went wrong.
|
This approach seems generally in the right direction. Presumably this change would also get ported to pypa/distutils and thus bundled with setuptools. If this provides the supported mechanism for distros to override prefixes and schemes, that's great. Does this change work with Python 3.6+ also? @FFY00 Is aiding in the effort, so I'd like his opinion too. |
Sorry, something went wrong.
|
This looks fine, I was planning doing something similar. I think with this merged and ported to pypa/distutils we are just missing a install scheme argument in the CLI and a mechanism for Python distributors to define their own site schemes (they would define a custom sysconfig scheme and that scheme would be used in the site module in addition to the default one). |
Sorry, something went wrong.
7876985 to
04346fd
Compare
April 2, 2021 08:48
Thanks for the review. It was a good idea to test this with older Pythons because it helped me to uncover a minor bug - the platlibdir variable is available only in Python >=3.9. Fixed. The latest version here works with Python 3.6+.
As far as I know from Fedora, the way we define our own schemas is by patching the distutils module. After this change, we are just going to patch sysconfig instead of distutils. But finding a better way to do the same in an easier way is definitely worth it. Given the positive reviews, @encukou, could you please merge this? Or is there anything more I can do? |
Sorry, something went wrong.
|
We need to get this in before beta 1. |
Sorry, something went wrong.
encukou
left a comment
There was a problem hiding this comment.
Looks mostly good! I have a few comments fixup and a question on the $platlibdir change, though.
Sorry, something went wrong.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot s390x SLES 3.x has failed when building commit 341e8a9. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/all/#builders/540/builds/1150 Failed tests:
Summary of the results of the build (if available): == Click to see traceback logsremote: Enumerating objects: 22, done.
remote: Counting objects: 4% (1/22)
remote: Counting objects: 9% (2/22)
remote: Counting objects: 13% (3/22)
remote: Counting objects: 18% (4/22)
remote: Counting objects: 22% (5/22)
remote: Counting objects: 27% (6/22)
remote: Counting objects: 31% (7/22)
remote: Counting objects: 36% (8/22)
remote: Counting objects: 40% (9/22)
remote: Counting objects: 45% (10/22)
remote: Counting objects: 50% (11/22)
remote: Counting objects: 54% (12/22)
remote: Counting objects: 59% (13/22)
remote: Counting objects: 63% (14/22)
remote: Counting objects: 68% (15/22)
remote: Counting objects: 72% (16/22)
remote: Counting objects: 77% (17/22)
remote: Counting objects: 81% (18/22)
remote: Counting objects: 86% (19/22)
remote: Counting objects: 90% (20/22)
remote: Counting objects: 95% (21/22)
remote: Counting objects: 100% (22/22)
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 8% (1/12)
remote: Compressing objects: 16% (2/12)
remote: Compressing objects: 25% (3/12)
remote: Compressing objects: 33% (4/12)
remote: Compressing objects: 41% (5/12)
remote: Compressing objects: 50% (6/12)
remote: Compressing objects: 58% (7/12)
remote: Compressing objects: 66% (8/12)
remote: Compressing objects: 75% (9/12)
remote: Compressing objects: 83% (10/12)
remote: Compressing objects: 91% (11/12)
remote: Compressing objects: 100% (12/12)
remote: Compressing objects: 100% (12/12), done.
remote: Total 12 (delta 10), reused 1 (delta 0), pack-reused 0
From https://github.com/python/cpython
* branch master -> FETCH_HEAD
Reset branch 'master'
test_winsound skipped -- No module named 'winsound'
test_ioctl skipped -- Unable to open /dev/tty
/tmp/tmppijdzxu7.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.util import byte_compile
/tmp/tmp89s3cse2.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.util import byte_compile
test_winreg skipped -- No module named 'winreg'
test_devpoll skipped -- test works only on Solaris OS family
test_winconsoleio skipped -- test only relevant on win32
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_kqueue skipped -- test works only on BSD
test_badargs (__main__.GeneralTest) ... ok
test_bound_methods (__main__.GeneralTest) ... ok
test_clear (__main__.GeneralTest) ... ok
test_exit (__main__.GeneralTest) ... ok
test_order (__main__.GeneralTest) ... ok
test_raise (__main__.GeneralTest) ... ok
test_raise_unnormalized (__main__.GeneralTest) ... ok
test_stress (__main__.GeneralTest) ... ok
test_unregister (__main__.GeneralTest) ... ok
----------------------------------------------------------------------
Ran 9 tests in 0.001s
OK
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/os.py'> has no attribute 'startfile'
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_nis skipped -- No module named 'nis'
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_msilib skipped -- No module named '_msi'
<string>:2: PendingDeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... ok
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok
----------------------------------------------------------------------
Ran 24 tests in 6.419s
OK (skipped=2)
Timeout (0:15:00)!
Thread 0x000003ffadf78720 (most recent call first):
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/selectors.py", line 469 in select
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 1845 in _run_once
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 595 in run_forever
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 628 in run_until_complete
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_subprocess.py", line 190 in test_terminate
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 549 in _callTestMethod
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 592 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 652 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/runner.py", line 176 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 959 in _run_suite
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 1082 in run_unittest
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 210 in _test_module
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 141 in _runtest
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 194 in runtest
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest_mp.py", line 81 in run_tests_worker
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 661 in _main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 641 in main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 719 in main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/regrtest.py", line 43 in _main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/regrtest.py", line 47 in <module>
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 86 in _run_code
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 196 in _run_module_as_main
Timeout (0:15:00)!
Thread 0x000003ffa1178720 (most recent call first):
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/selectors.py", line 469 in select
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 1845 in _run_once
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 595 in run_forever
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 628 in run_until_complete
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_subprocess.py", line 190 in test_terminate
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 549 in _callTestMethod
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 592 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 652 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/runner.py", line 176 in run
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 959 in _run_suite
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 1082 in run_unittest
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 210 in _test_module
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 154 in _runtest
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 194 in runtest
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 321 in rerun_failed_tests
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 698 in _main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 641 in main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 719 in main
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/__main__.py", line 2 in <module>
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 86 in _run_code
File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 196 in _run_module_as_main
make: *** [Makefile:1247: buildbottest] Error 1
Cannot open file '/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/test-results.xml' for upload |
Sorry, something went wrong.
|
The buildbot test failed on timeout which I believe has nothing to do with this PR. |
Sorry, something went wrong.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
pythonGH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
GH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
pythonGH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
(cherry picked from commit 563bd5a)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
GH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
(cherry picked from commit 563bd5a)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
With this patch
distutils.command.install.INSTALL_SCHEMESare loaded fromsysconfig._INSTALL_SCHEMES.The
distutilsmodule is deprecated and will be removed in 3.12 (PEP 632). This change makes thesysconfig._INSTALL_SCHEMESthe single point o truth for install schemes while keepingdistutils.command.install.INSTALL_SCHEMESexactly the same. If we, during the transition to the sysconfig, change something, this makes sure that it propagates also to distutils until the module gets removed.Moreover, as discussed there, Linux distros need to patch distutils/sysconfig to make sure the packages will land in proper locations. This patch makes it easier because it leaves only one location where install schemes are defined which is much easier to patch/adjust. If accepted, a downstream patch in Fedora might look like this.
The implementation is slightly different than the plan but I think it's the easiest way how to do it and it also makes the downstream patch simple, flexible and easy to maintain.
It's also necessary to implement this before setuptools starts bundling the distutils module so the default install schemes stay in the standard library.
The removed code from
sysconfigdoes not seem to have any negative effect because, honestly, it seems that nothing actually uses the install schemes from sysconfig at all. There were many big changes in these modules where they were trying to include packaging in stdlib and then reverted that. Also, the test of distutils install command does not count with the different locations which is good evidence that the reason to have this piece of code is no longer valid.Cc: @hroncok @encukou @jaraco @pradyunsg
https://bugs.python.org/issue41282