bpo-27640: Add --disable-test-modules configure option#23886

pxinwr

Test modules under Lib directory are used for internal CPython regression testing. Shipping them in formal release is not necessary. So to reduce package size and build and installation time, we should provide an option in configure to disable them.

https://bugs.python.org/issue27640

doko42

Parts of the testsuite are used by third party packages. Even if you don't want to install test modules, you should install those.
This is what I identify as part of the test framework:

           $(scriptdir)/test/{libregrtest,support} \
            $(scriptdir)/test/{ann_module,ann_module2,ann_module3}.py \
            $(scriptdir)/test/{regrtest,test_support,__init__,__main__}.py

However that list might be incomplete.

For packaging purposes I think it's better to remove those files which you don't want to ship after the installation.

doko42

Parts of the testsuite are used by third party packages. Even if you don't want to install test modules, you should install those.

This is what I identify as part of the test framework:

test/{libregrtest,support}
test/{ann_module,ann_module2,ann_module3}.py
test/{regrtest,test_support,init,main}.py

However that list might be incomplete.

For packaging purposes I think it's better to remove those files which you don't want to ship after the installation.

pxinwr

@doko42 Thanks for your review. After I installed the Python3.8 distribution for Ubuntu, it indeed has the regression test framework shipped but with all other test stuffs removed. So always shipping test framework skeleton regardless of the option value of --enable-test-modules could be a good idea. Could we do the same thing like Ubuntu does as shown below?

Under the directory /usr/lib/python3.8/test, Ubuntu ships:
__init__.py libregrtest __main__.py regrtest.py support test_support.py

In addition, Ubuntu doesn't ship test/{ann_module,ann_module2,ann_module3}.py. How are these used by 3rd-party?

vstinner

https://bugs.python.org/issue31904 is a meta-issue about VxWorks, but I would prefer to have a dedicated issue for this change.

The good news is that it already exists: https://bugs.python.org/issue27640 Please reuse this BPO.

bedevere-bot

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vstinner

The buildroot project has the following patch on Python which adds --disable-test-modules configure option:
https://git.buildroot.org/buildroot/tree/package/python3/0013-Add-an-option-to-disable-installation-of-test-module.patch

Co-authored-by: Victor Stinner <vstinner@python.org>

doko42

https://bugs.debian.org/944303 showed that python-typing-extensions uses the ann* modules. Now that's integrated in upstream cpython, but still shipped on PyPi: https://pypi.org/project/typing-extensions/. So yes, I think these could be removed as well.

Please note that Debian/Ubuntu packaging doesn't remove any of these files, but just ships them in a separate binary package, e.g. libpython3.8-testsuite.

pxinwr

@vstinner Regarding the regression test framework(see below for files and directories) what is your opinion? Keeping it or removing it entirely when disabling test suites? Looks buildroot removes it while Ubuntu keeps it. If we decide keeping it, some of the test extension modules(_testcapi and _testinternalcapi) have to be kept as well because test.support needs to import them.

__init__.py libregrtest __main__.py regrtest.py support test_support.py

pxinwr

I have made the requested changes; please review again

bedevere-bot

Thanks for making the requested changes!

@vstinner: please review the changes made to this pull request.

vstinner

The Fedora package put the following files/directories in the python3-test binary RPM:

%if %{without flatpackage}
%files -n %{pkgname}-test
%endif

%{pylibdir}/ctypes/test
%{pylibdir}/distutils/tests
%{pylibdir}/sqlite3/test
%{pylibdir}/test
%{dynload_dir}/_ctypes_test.%{SOABI_optimized}.so
%{dynload_dir}/_testbuffer.%{SOABI_optimized}.so
%{dynload_dir}/_testcapi.%{SOABI_optimized}.so
%{dynload_dir}/_testimportmultiple.%{SOABI_optimized}.so
%{dynload_dir}/_testinternalcapi.%{SOABI_optimized}.so
%{dynload_dir}/_testmultiphase.%{SOABI_optimized}.so
%{dynload_dir}/_xxtestfuzz.%{SOABI_optimized}.so
%{pylibdir}/lib2to3/tests
%{pylibdir}/tkinter/test
%{pylibdir}/unittest/test

https://src.fedoraproject.org/rpms/python3.10/blob/master/f/python3.10.spec#_1429

vstinner

With the PR and --disable-test-modules, the following directories (and sub-directories) are no longer installed:

ctypes/test/
distutils/tests/
idlelib/idle_test/
lib2to3/tests/
sqlite3/test/
test/
tkinter/test/
unittest/test/

With the PR and --disable-test-modules, the following files of lib-dynload/ are no longer installed:

_ctypes_test.cpython-310-x86_64-linux-gnu.so
_testbuffer.cpython-310-x86_64-linux-gnu.so
_testcapi.cpython-310-x86_64-linux-gnu.so
_testimportmultiple.cpython-310-x86_64-linux-gnu.so
_testinternalcapi.cpython-310-x86_64-linux-gnu.so
_testmultiphase.cpython-310-x86_64-linux-gnu.so
_xxtestfuzz.cpython-310-x86_64-linux-gnu.so

Install in /opt/py310 without tests:

./configure --disable-test-modules --prefix /opt/py310test CFLAGS="-O0" --with-platlibdir=lib64 && make && make install

Install in /opt/py310test with tests:

./configure --prefix /opt/py310test CFLAGS="-O0" --with-platlibdir=lib64 && make && make install

Remove .pyc files:

find /opt/py310 -name "*.pyc" -delete
find /opt/py310test -name "*.pyc" -delete

Compare the content of the two directories:

(cd /opt/py310test/lib64/python3.10/; find)|sort > A
(cd /opt/py310/lib64/python3.10/; find)|sort > B
diff -u B A > pr23886_files.diff

I atttached the full pr23886_files.diff to https://bugs.python.org/issue27640

vstinner

Good: this PR does not remove any file/directory by default (compared to the master branch). It works as expected.

$ git switch master
$ ./configure --prefix /opt/py310master CFLAGS="-O0" --with-platlibdir=lib64 && make && make install 
$ find /opt/py310master -name "*.pyc" -delete

$ git switch pr/23886
$ ./configure --prefix /opt/py310pr CFLAGS="-O0" --with-platlibdir=lib64 && make && make install 
$ find /opt/py310pr -name "*.pyc" -delete

$ (cd /opt/py310master/; find)|sort > A; (cd /opt/py310pr/; find)|sort > B
$ diff -u A B

# diff output is empty: that's good! there is no change

vstinner

On Fedora, the python3 package does not contain the test module. You must install python3-test.

The test module should not be used by third party projects. The documentation starts with the note:

The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.

https://docs.python.org/dev/library/test.html

vstinner

@doko42:

https://bugs.debian.org/944303 showed that python-typing-extensions uses the ann* modules. Now that's integrated in upstream cpython, but still shipped on PyPi: https://pypi.org/project/typing-extensions/. So yes, I think these could be removed as well.

Please note that Debian/Ubuntu packaging doesn't remove any of these files, but just ships them in a separate binary package, e.g. libpython3.8-testsuite.

On Debian, Ubuntu or Fedora, it's just a matter of putting the right dependency on the libpython3.8-testsuite (python3-test on Fedora) package.

@pxinwr:

@vstinner Regarding the regression test framework(see below for files and directories) what is your opinion? Keeping it or removing it entirely when disabling test suites? Looks buildroot removes it while Ubuntu keeps it.

This PR is not intended for general Linux distributions like Debian or Fedora, but specialized embedded devices. Xavier created https://bugs.python.org/issue27640 when he worked on putting Python on Android. The buildroot is also intended to be used to produce an image for an embedded devices. The use case is to ship an image for end users who will not run any test but use a "finished" product. For developers, well, just don't use --disable-test-modules.

Co-authored-by: Victor Stinner <vstinner@python.org>

Co-Authored-By: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

Co-authored-by: Victor Stinner <vstinner@python.org>

vstinner

I tested again the PR, at commit 9e5dada. I rebased locally the PR on master. The PR works as expected!

Install Python 3 times:

master
PR with tests
PR without tests

# master with tests
$ git switch master
$ git clean -fdx
$ ./configure --prefix /opt/py310_master CFLAGS="-O0"
$ make && make install

# PR with tests
$ git switch pr/23886
$ git clean -fdx
$ ./configure --disable-test-modules --prefix /opt/py310_notest CFLAGS="-O0"
$ make && make install

# PR without tests
$ git switch pr/23886
$ git clean -fdx
$ ./configure --prefix /opt/py310_pr CFLAGS="-O0"
$ make && make install

First test: ensure that the PR still copy exactly the same files: good, there is zero difference!

$ (cd /opt/py310_master; find|sort) > files-master
$ (cd /opt/py310_pr; find|sort) > files-pr-with-tests
$ diff -u files-master files-pr-with-tests 

# diff output is empty: good! there is no change

Second test: check that --disable-test-modules don't install tests. Good! Test C extension are not built nor installed and test subdirectories are not installed, as expected.

$ (cd /opt/py310_master; find|sort) > files-master
$ (cd /opt/py310_notest; find|sort) > files-pr-without-tests

$ (cd /opt/py310_master; find -type d|sort) > dirs-master
$ (cd /opt/py310_notest; find -type d|sort) > dirs-pr-without-tests

# Removed files
$ diff -u files-master files-pr-without-tests |grep '^-.*so$'
-./lib/python3.10/lib-dynload/_ctypes_test.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testbuffer.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testcapi.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testimportmultiple.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testinternalcapi.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testmultiphase.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_xxtestfuzz.cpython-310-x86_64-linux-gnu.so

# Removed directories
$ diff -u dirs-master dirs-pr-without-tests |grep ^-|grep -v __pycache__
-./lib/python3.10/ctypes/test
-./lib/python3.10/distutils/tests
-./lib/python3.10/idlelib/idle_test
-./lib/python3.10/lib2to3/tests
-./lib/python3.10/lib2to3/tests/data
-./lib/python3.10/lib2to3/tests/data/fixers
-./lib/python3.10/lib2to3/tests/data/fixers/myfixes
-./lib/python3.10/sqlite3/test
-./lib/python3.10/test
-./lib/python3.10/test/audiodata
-./lib/python3.10/test/capath
-./lib/python3.10/test/cjkencodings
-./lib/python3.10/test/data
-./lib/python3.10/test/decimaltestdata
-./lib/python3.10/test/dtracedata
-./lib/python3.10/test/encoded_modules
-./lib/python3.10/test/imghdrdata
-./lib/python3.10/test/libregrtest
-./lib/python3.10/test/sndhdrdata
-./lib/python3.10/test/subprocessdata
-./lib/python3.10/test/support
-./lib/python3.10/test/test_asyncio
-./lib/python3.10/test/test_email
-./lib/python3.10/test/test_email/data
-./lib/python3.10/test/test_import
-./lib/python3.10/test/test_import/data
-./lib/python3.10/test/test_import/data/circular_imports
-./lib/python3.10/test/test_import/data/circular_imports/subpkg
-./lib/python3.10/test/test_import/data/package
-./lib/python3.10/test/test_import/data/package2
-./lib/python3.10/test/test_import/data/unwritable
-./lib/python3.10/test/test_importlib
-./lib/python3.10/test/test_importlib/builtin
-./lib/python3.10/test/test_importlib/data
-./lib/python3.10/test/test_importlib/data01
-./lib/python3.10/test/test_importlib/data01/subdirectory
-./lib/python3.10/test/test_importlib/data02
-./lib/python3.10/test/test_importlib/data02/one
-./lib/python3.10/test/test_importlib/data02/two
-./lib/python3.10/test/test_importlib/data03
-./lib/python3.10/test/test_importlib/data03/namespace
-./lib/python3.10/test/test_importlib/data03/namespace/portion1
-./lib/python3.10/test/test_importlib/data03/namespace/portion2
-./lib/python3.10/test/test_importlib/extension
-./lib/python3.10/test/test_importlib/frozen
-./lib/python3.10/test/test_importlib/import_
-./lib/python3.10/test/test_importlib/namespace_pkgs
-./lib/python3.10/test/test_importlib/namespace_pkgs/both_portions
-./lib/python3.10/test/test_importlib/namespace_pkgs/both_portions/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/module_and_namespace_package
-./lib/python3.10/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test
-./lib/python3.10/test/test_importlib/namespace_pkgs/not_a_namespace_pkg
-./lib/python3.10/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion1
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion1/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion2
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion2/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/project1
-./lib/python3.10/test/test_importlib/namespace_pkgs/project1/parent
-./lib/python3.10/test/test_importlib/namespace_pkgs/project1/parent/child
-./lib/python3.10/test/test_importlib/namespace_pkgs/project2
-./lib/python3.10/test/test_importlib/namespace_pkgs/project2/parent
-./lib/python3.10/test/test_importlib/namespace_pkgs/project2/parent/child
-./lib/python3.10/test/test_importlib/namespace_pkgs/project3
-./lib/python3.10/test/test_importlib/namespace_pkgs/project3/parent
-./lib/python3.10/test/test_importlib/namespace_pkgs/project3/parent/child
-./lib/python3.10/test/test_importlib/source
-./lib/python3.10/test/test_importlib/zipdata01
-./lib/python3.10/test/test_importlib/zipdata02
-./lib/python3.10/test/test_json
-./lib/python3.10/test/test_peg_generator
-./lib/python3.10/test/test_tools
-./lib/python3.10/test/test_warnings
-./lib/python3.10/test/test_warnings/data
-./lib/python3.10/test/test_zoneinfo
-./lib/python3.10/test/test_zoneinfo/data
-./lib/python3.10/test/tracedmodules
-./lib/python3.10/test/xmltestdata
-./lib/python3.10/test/xmltestdata/c14n-20
-./lib/python3.10/test/ziptestdata
-./lib/python3.10/tkinter/test
-./lib/python3.10/tkinter/test/test_tkinter
-./lib/python3.10/tkinter/test/test_ttk
-./lib/python3.10/unittest/test
-./lib/python3.10/unittest/test/testmock

vstinner

Thank you @pxinwr for your multiple updates, the final PR is much better. The change is now properly documented (NEWS entry, What's New in Python 3.10 entry), the option is properly documented directly in configure (it says exactly what it does), it's better than Xavier's initial patch and the Thoma's buildbroot patch since it also doesn't build test extensions. And overall, I think that it's a reasonable build option, it makes sense to not install tests on a small "embedded device".

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora 3.x has failed when building commit 277ce30.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/223/builds/528) and take a look at the build logs.
Check if the failure is related to this commit (277ce30) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/223/builds/528

Failed tests:

test_asyncio

Failed subtests:

test_start_new_session - test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/asyncio/unix_events.py", line 1302, in _do_waitpid
    loop, callback, args = self._callbacks.pop(pid)
KeyError: 3703654


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/test/test_asyncio/test_subprocess.py", line 169, in test_start_new_session
    self.assertEqual(exitcode, 8)
AssertionError: 255 != 8

Added --disable-test-modules option to the configure script: don't build nor install test modules. Patch by Xavier de Gaye, Thomas Petazzoni and Peixing Xin. Co-Authored-By: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com>

Add --enable-test-modules option for configure

9cc3b30

the-knights-who-say-ni added the CLA signed label Dec 22, 2020

bedevere-bot added the awaiting review label Dec 22, 2020

pxinwr changed the title ~~bpo-41443: Add --enable-test-modules option for configure~~ Dec 22, 2020

fix no result issues when checking

9a2fcad

doko42 self-requested a review December 24, 2020 10:18

doko42 reviewed Dec 24, 2020

View reviewed changes

vstinner requested changes Dec 25, 2020

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Dec 25, 2020

Update setup.py …

3031f42

Co-authored-by: Victor Stinner <vstinner@python.org>

pxinwr added 5 commits December 28, 2020 21:31

rename news file

c4f193f

add info in whatsnew

96bd20c

elaborate the --enable-test-modules in comments

c655075

sort the directories listed in LIBSUBDIRS and TESTSUBDIRS

cd21bd6

regenerate configure

f48d197

pxinwr changed the title ~~bpo-31904: Add --enable-test-modules option for configure~~ Dec 28, 2020

update news

2d5f0e3

bedevere-bot added awaiting change review and removed awaiting changes labels Dec 29, 2020

bedevere-bot requested a review from vstinner December 29, 2020 06:18

vstinner reviewed Dec 29, 2020

View reviewed changes

vstinner reviewed Dec 30, 2020

View reviewed changes

vstinner changed the title ~~bpo-27640: Add --enable-test-modules option for configure~~ Dec 30, 2020

vstinner changed the title ~~bpo-27640: Add --disable-test-modules option for configure~~ Dec 30, 2020

pxinwr and others added 5 commits December 30, 2020 14:15

Improve Doc/whatsnew/3.10.rst …

17c08dd

Co-authored-by: Victor Stinner <vstinner@python.org>

credit Xavier de Gaye and Thomas Petazzoni in Doc/whatsnew/3.10.rst …

cdd5fa6

Co-Authored-By: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

improve help string for the added option …

23daf7a

Co-authored-by: Victor Stinner <vstinner@python.org>

change the help and message for the new option

90e4114

credit Xavier and Thomas in news

9e5dada

vstinner merged commit 277ce30 into python:master Dec 30, 2020

bedevere-bot removed the awaiting change review label Dec 30, 2020

pxinwr deleted the fix-issue-31904-distest branch May 7, 2021 07:42

kuhlenough mannequin mentioned this pull request Jan 12, 2024

Python should support VxWorks RTOS #76085

Closed

Conversation

pxinwr commented Dec 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doko42 commented Dec 24, 2020

Uh oh!

doko42 left a comment

Choose a reason for hiding this comment

Uh oh!

pxinwr commented Dec 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Dec 25, 2020

Uh oh!

vstinner commented Dec 25, 2020

Uh oh!

doko42 commented Dec 28, 2020

Uh oh!

pxinwr commented Dec 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pxinwr commented Dec 29, 2020

Uh oh!

bedevere-bot commented Dec 29, 2020

Uh oh!

vstinner commented Dec 29, 2020

Uh oh!

vstinner commented Dec 29, 2020

Uh oh!

vstinner commented Dec 30, 2020

Uh oh!

vstinner commented Dec 30, 2020

Uh oh!

vstinner commented Dec 30, 2020

Uh oh!

vstinner commented Dec 30, 2020

Uh oh!

vstinner commented Dec 30, 2020

Uh oh!

bedevere-bot commented Dec 30, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pxinwr commented Dec 22, 2020 •

edited

Loading

pxinwr commented Dec 25, 2020 •

edited

Loading

pxinwr commented Dec 29, 2020 •

edited

Loading