◐ Shell
reader mode source ↗
Skip to content

gh-92584: Remove the distutils package#99061

Merged
vstinner merged 2 commits into
python:mainfrom
vstinner:remove_distutils
Nov 3, 2022
Merged

gh-92584: Remove the distutils package#99061
vstinner merged 2 commits into
python:mainfrom
vstinner:remove_distutils

Conversation

@vstinner

@vstinner vstinner commented Nov 3, 2022 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

Remove the distutils package. It was deprecated in Python 3.10 by PEP 632 "Deprecate distutils module". For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils.

  • Remove Lib/distutils/ directory
  • Remove Doc/installing/ directory
  • Remove test_distutils
  • Remove references to distutils
  • Skip test_check_c_globals and test_peg_generator since they use distutils

@vstinner vstinner requested review from a team, lysnikolaou and pablogsal as code owners November 3, 2022 14:33
@bedevere-bot bedevere-bot mentioned this pull request Nov 3, 2022
Closed
@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

There are still many references to distutils. I prefer to remove/update them in separated PRs to keep this PR as small as possible.

Doc/distributing/index.rst:7::Email: distutils-sig@python.org
Doc/distributing/index.rst:44:* ``distutils`` is the original build and distribution system first added
Doc/distributing/index.rst:45:  to the Python standard library in 1998. While direct use of ``distutils``
Doc/distributing/index.rst:51:* `setuptools`_ is a (largely) drop-in replacement for ``distutils`` first
Doc/distributing/index.rst:53:  ``distutils`` tools was the ability to declare dependencies on other
Doc/distributing/index.rst:55:  alternative to ``distutils`` that offers consistent support for more
Doc/distributing/index.rst:58:  command to ``distutils``/`setuptools`_. This produces a cross platform

Doc/extending/building.rst:53:Python 3.12 and newer no longer come with distutils. Please refer to the
Doc/extending/newtypes_tutorial.rst:203:   from distutils.core import setup, Extension
Doc/extending/newtypes_tutorial.rst:223:   While this documentation showcases the standard :mod:`distutils` module
Doc/extending/newtypes_tutorial.rst:530:   from distutils.core import setup, Extension
Doc/extending/windows.rst:17:Module authors are encouraged to use the distutils approach for building

Doc/includes/setup.py:1:from distutils.core import setup, Extension
Doc/includes/test.py:192:from distutils.util import get_platform

Doc/install/index.rst:15:   The entire ``distutils`` package has been deprecated and will be
Doc/install/index.rst:18:   :ref:`What's New <distutils-deprecated>` entry for more information.
Doc/install/index.rst:47:In Python 2.0, the ``distutils`` API was first added to the standard library.
Doc/install/index.rst:56:using ``distutils`` directly.
Doc/install/index.rst:634:   that change the installation path will be ignored from all distutils configuration
Doc/install/index.rst:748:| system       | :file:`{prefix}/lib/python{ver}/distutils/distutils.cfg` | \(1)  |
Doc/install/index.rst:750:| personal     | :file:`$HOME/.pydistutils.cfg`                           | \(2)  |
Doc/install/index.rst:760:| system       | :file:`{prefix}\\Lib\\distutils\\distutils.cfg` | \(4)  |
Doc/install/index.rst:762:| personal     | :file:`%HOME%\\pydistutils.cfg`                 | \(5)  |
Doc/install/index.rst:776:   :file:`{prefix}/lib/python1.5/site-packages/distutils`, so the system
Doc/install/index.rst:791:   :file:`C:\\Python\\Lib\\distutils\\distutils.cfg`. Under Python 1.5.2, the
Doc/install/index.rst:794:   :file:`C:\\Program Files\\Python\\distutils\\distutils.cfg` in a standard Python

Doc/library/importlib.metadata.rst:90:``console_scripts``, ``distutils.commands`` and others.  Each group contains a
Doc/library/importlib.metadata.rst:130:    ['console_scripts', 'distutils.commands', 'distutils.setup_keywords', 'egg_info.writers', 'setuptools.installation']

Doc/library/sysconfig.rst:72:Every new component that is installed using :mod:`distutils` or a

Doc/library/zipapp.rst:351:command line tools, or you can take advantage of the fact that distutils
Doc/library/zipapp.rst:354:   >>> from distutils.ccompiler import new_compiler
Doc/library/zipapp.rst:355:   >>> import distutils.sysconfig
Doc/library/zipapp.rst:364:   >>>     cc.add_include_dir(distutils.sysconfig.get_python_inc())

Doc/using/configure.rst:681:the :mod:`distutils` module.
Doc/using/configure.rst:754:   distutils :envvar:`CFLAGS` once Python is installed (:issue:`21121`).
Doc/using/configure.rst:877:   the distutils :envvar:`LDFLAGS` once Python is installed (:issue:`35257`).

Doc/using/mac.rst:130:* Packages can be installed via the standard Python distutils mode (``python
Doc/using/windows.rst:1208:`cx_Freeze <https://cx-freeze.readthedocs.io/en/latest/>`_ is a ``distutils``

Lib/_aix_support.py:9:    # _aix_support is used in distutils by setup.py to build C extensions,

Lib/_osx_support.py:91:    # the build of Python itself (distutils is used to build standard library
Lib/_osx_support.py:396:        # see also distutils.sysconfig
Lib/_osx_support.py:452:    distutils.sysconfig.get_config_vars() is first
Lib/_osx_support.py:458:    requested (in distutils.sysconfig.customize_compiler).
Lib/_osx_support.py:460:    Currently called from distutils.sysconfig
Lib/_osx_support.py:484:    in distutils.sysconfig.customize_compiler.
Lib/_osx_support.py:501:    # called from get_platform() in sysconfig and distutils.util
Lib/_osx_support.py:504:    # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set
Lib/_osx_support.py:519:        # Otherwise, distutils may consider this a cross-compiling

Lib/test/test__osx_support.py:316:        # indirectly by test_sysconfig and test_distutils

Lib/site.py:84:# for distutils.commands.install

Lib/sysconfig.py:431:                # This is disabled when called from distutils.sysconfig

Lib/test/support/__init__.py:1853:    # TODO (PEP 632): alternate check without using distutils
Lib/test/support/__init__.py:1854:    from distutils import ccompiler, sysconfig, spawn, errors

Lib/test/test_check_c_globals.py:6:# TODO: gh-92584: c-analyzer uses distutils which was removed in Python 3.12
Lib/test/test_check_c_globals.py:7:raise unittest.SkipTest("distutils has been removed in Python 3.12")

Lib/test/test_importlib/test_windows.py:21:    # Port of distutils.util.get_platform().

Lib/test/test_peg_generator/__init__.py:6:# TODO: gh-92584: peg_generator uses distutils which was removed in Python 3.12
Lib/test/test_peg_generator/__init__.py:7:raise unittest.SkipTest("distutils has been removed in Python 3.12")

Mac/BuildScript/build-installer.py:1332:    # distutils.tests.test_sysconfig.SysconfigTestCase tests that
Mac/BuildScript/build-installer.py:1334:    # distutils.sysconfig is exactly the same for both CFLAGS and

Mac/PythonLauncher/Makefile.in:18:# by distutils

Makefile.pre.in:86:# Use it when a compiler flag should _not_ be part of the distutils CFLAGS
Makefile.pre.in:90:# Use it when a linker flag should _not_ be part of the distutils LDFLAGS
Makefile.pre.in:187:# by distutils. The export statement is needed to ensure that the

PC/pyconfig.h:276:                        generally taken care of by distutils.) */

Tools/c-analyzer/c_parser/preprocessor/__init__.py:228:    # matching distutils.ccompiler.compiler_class:
Tools/c-analyzer/c_parser/preprocessor/common.py:2:import distutils.ccompiler
Tools/c-analyzer/c_parser/preprocessor/common.py:80:    compiler = distutils.ccompiler.new_compiler(

Tools/peg_generator/pegen/build.py:46:    of distutils (this is useful in case you want to use a temporary directory).
Tools/peg_generator/pegen/build.py:52:    import distutils.log
Tools/peg_generator/pegen/build.py:53:    from distutils.core import Distribution, Extension
Tools/peg_generator/pegen/build.py:54:    from distutils.tests.support import fixup_build_ext  # type: ignore
Tools/peg_generator/pegen/build.py:56:    from distutils.ccompiler import new_compiler
Tools/peg_generator/pegen/build.py:57:    from distutils.dep_util import newer_group
Tools/peg_generator/pegen/build.py:58:    from distutils.sysconfig import customize_compiler
Tools/peg_generator/pegen/build.py:61:        distutils.log.set_threshold(distutils.log.DEBUG)

configure.ac:727:# it may influence the way we can build extensions, so distutils
configure.ac:905:  By default, distutils will build C++ extension modules with "$CXX".
configure.ac:1375:# distutils.unixccompiler to know if it should add --enable-new-dtags
configure.ac:1732:  # compile working code using it and both test_distutils and test_gdb are
configure.ac:2450:        # component of the string returned by distutils.get_platform().

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

The doc build fails when I just removed Doc/installing/index.rst. I reverted this change.

Warning: /home/runner/work/cpython/cpython/Doc/distributing/index.rst:19: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/install/index.rst:23: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/install/index.rst:58: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/library/ensurepip.rst:34: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/tutorial/venv.rst:105: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/tutorial/venv.rst:207: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/tutorial/whatnow.rst:24: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/whatsnew/2.7.rst:2690: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/whatsnew/3.4.rst:240: WARNING: undefined label: installing-index

It should be removed in a separated PR.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

Optional Windows (x86) CI job failed, I re-run the job ("2 re-run tests: test_asyncio test_threading"). It's just a random error, unrelated to this PR.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

My colleague @hroncok is already preparing Python packages in Fedora for this removal: Help needed triaging build failures without distutils. 250 packages are impacted: a build dependency to setuptools should be added to get distutils.

@vstinner vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 3, 2022
@bedevere-bot

bedevere-bot commented Nov 3, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 5334545 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 3, 2022
@hroncok

hroncok commented Nov 3, 2022

Copy link
Copy Markdown
Contributor

My colleague @hroncok is already preparing Python packages in Fedora for this removal: Help needed triaging build failures without distutils. 250 packages are impacted: a build dependency to setuptools should be added to get distutils.

A handful of packages also falls to build with setutpools, because the distutils module from setuptools is not backward compatible with distutils.

zware
zware approved these changes Nov 3, 2022
View reviewed changes

@zware zware left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM (though there's two more lines that can be removed in PC/layout/support).

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

A handful of packages also falls to build with setutpools, because the distutils module from setuptools is not backward compatible with distutils.

That's... very scary :-(

@CAM-Gerlach

CAM-Gerlach commented Nov 3, 2022

Copy link
Copy Markdown
Member

The naming is confusing, but Doc/installing, which is stated to be removed in the OP (but later the docs changes were decided for a separate PR) is not the distutils docs, its the modern(ish), thin-shim guide on installing things with Python, and only mentions distutils one place, in the context of describing it as a deprecated legacy module. So that one sentence should be updated, but the rest is unrelated to distutils and its removal, if desired, is seems out of scope here.

Meanwhile, Doc/install is the actual distutils documentation, which should presumably be removed (it can always be accessed in older Python versions) but is not listed above or removed by this PR.

@vstinner Would you like me to help take care of the docs changes/removals in a followup PR, since I'm fairly familiar with them? They're also somewhat covered by a separate issue, #94249

@vstinner
gh-92584: Remove the distutils package …
a8923b4 
Remove the distutils package. It was deprecated in Python 3.10 by PEP
632 "Deprecate distutils module". For projects still using distutils
and cannot be updated to something else, the setuptools project can
be installed: it still provides distutils.

* Remove Lib/distutils/ directory
* Remove test_distutils
* Remove references to distutils
* Skip test_check_c_globals and test_peg_generator since they use
  distutils
@vstinner
Update PC/layout/
eca9891
@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

The test suite passed on most buildbots: great! I will not wait for the few remaining buildbots (like Refleaks buildbots, removing pure Python code is unlikely to introduce such leak).

I addressed @zware's review in a second commit. I will merge my PR as soon as the CI pass on it.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

@vstinner Would you like me to help take care of the docs changes/removals in a followup PR

Yes! Help on updating the doc is welcomed. There is a lot of documentation mentioning distutils, and deciding if the doc must be removed or not is not easy.

@vstinner vstinner merged commit 0faa0ba into python:main Nov 3, 2022
@vstinner vstinner deleted the remove_distutils branch November 3, 2022 18:27
@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

Ok, I removed the distutils packages. Now test_check_c_globals and test_peg_generator should be updated to no longer use distutils. And remaining references to distutils #99061 (comment) should be removed. @CAM-Gerlach is volunteer to handle the Doc/ part.

@CAM-Gerlach CAM-Gerlach mentioned this pull request Nov 3, 2022
Closed
9 tasks
@rgommers rgommers mentioned this pull request Nov 6, 2022
Closed
@hugovk hugovk mentioned this pull request Nov 29, 2022
Closed
@astrojuanlu astrojuanlu mentioned this pull request Jan 31, 2023
Merged
5 tasks
@fperrin fperrin mentioned this pull request Jul 24, 2023
Merged
@gaborbernat gaborbernat mentioned this pull request Jun 3, 2026
Open
@coderabbitai coderabbitai Bot mentioned this pull request Jun 11, 2026
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou encukou left review comments

@zware zware zware approved these changes

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@vstinner @bedevere-bot @hroncok @CAM-Gerlach @encukou @zware