bpo-45950: Introduce Bootstrap Python again#29859

tiran

The build system now uses a :program:_bootstrap_python interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:_bootstrap_python and :program:_freeze_module are no longer
build with LTO.

Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.

https://bugs.python.org/issue45950

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @tiran for commit b757d5bd526cfffeb134eb6744ac86c561e61e9d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @tiran for commit fa4cd7fc9f3763ff1309d8bfbc97853223f17239 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

gvanrossum

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

vstinner

it's just an extra link step

When LTO is used, a link step can be quite slow. I noticed that when there was a second _bootstrap_python binary. I didn't check if _testembed has a similar issue or not. I'm mostly developing Python without --enable-shared: Python is linked "statically" in all built executables.

vstinner

Maybe LTO flags can be skipped on these less important binaries.

vstinner

FYI the revert #29717 fixed an issue with out of tree build. Please try to make sure that this bug is not reintroduced: https://bugs.python.org/issue45866#msg406764

tiran

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

We need some extra logic in configure.ac and Makefile anyway. The _bootstrap_python approach is incompatible with cross compiling. When you cross compile for a different architecture or platform, then any compiled code is not executable on the build host.

The approach also speeds up local development as long as you have a recent Python installation on your computer. With _bootstrap_python almost every code file gets rebuild whenever a core file changes. Scratch that, the Program/_freeze_module dependency triggers a partial rebuild.

I'll come up with a better way to test _buildbot_python.

gvanrossum

I am in favor of this. Thanks for doing this!

hroncok

I'll try to build this as RPM in Fedora to see if there are some problems with our bootstrapped OOT build.

hroncok

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

...
gcc -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g   -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g  -fno-lto -o Programs/_freeze_module Programs/_freeze_module.o Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o   Python/pydtrace.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o -ldl                         -lm 
/usr/bin/ld: Programs/_freeze_module.o: in function `runtime_init':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:44: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o: in function `_Py_DECREF':
/builddir/build/BUILD/cpython-bootstrap_again/Include/object.h:493: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:(.data.rel.ro+0x0): undefined reference to `__gcov_indirect_call'
/usr/bin/ld: Programs/_freeze_module.o: in function `main':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:191: undefined reference to `__gcov_indirect_call'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_indirect_call_profiler_v4'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:211: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: more undefined references to `__gcov_time_profiler_counter' follow
...

tiran

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

tiran

OOT build with optimizations is passing for me on F35:

../../configure -C --enable-optimizations``
make

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @tiran for commit 02da340e62db33a12aad1b03adf83a488e0255fb 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

hroncok

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

That helped.

tiran

Rebased on top of #29902

The build system now uses a `_bootstrap_python` interpreter for freezing and deepfreezing again. Cross building depends on a build Python interpreter. `_bootstrap_python` and `_freeze_module` are no longer compiled with LTO. Link time optimization is very slow.

bedevere-bot

@tiran: Please replace # with GH- in the commit message next time. Thanks!

the-knights-who-say-ni added the CLA signed label Nov 30, 2021

bedevere-bot added the awaiting core review label Nov 30, 2021

tiran force-pushed the bootstrap_again branch from fe57850 to b757d5b Compare November 30, 2021 10:05

vstinner reviewed Nov 30, 2021

View reviewed changes

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

tiran force-pushed the bootstrap_again branch from b757d5b to fa4cd7f Compare November 30, 2021 11:07

tiran added the 🔨 test-with-buildbots label Nov 30, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

gvanrossum self-requested a review November 30, 2021 15:05

tiran force-pushed the bootstrap_again branch 3 times, most recently from 757a652 to 551fb30 Compare December 1, 2021 12:03

gvanrossum approved these changes Dec 1, 2021

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Dec 1, 2021

tiran force-pushed the bootstrap_again branch from 551fb30 to cfe4551 Compare December 1, 2021 16:28

tiran changed the title ~~Bootstrap Python again~~ Dec 1, 2021

tiran marked this pull request as ready for review December 1, 2021 16:29

tiran force-pushed the bootstrap_again branch from cfe4551 to 02da340 Compare December 2, 2021 15:28

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2021

tiran mentioned this pull request Dec 2, 2021

bpo-45949: Pure Python freeze module for cross builds (GH-29899) #29899

Merged

tiran force-pushed the bootstrap_again branch from 02da340 to 252b53a Compare December 3, 2021 10:48

tiran requested a review from a team as a code owner December 3, 2021 10:48

tiran force-pushed the bootstrap_again branch from 252b53a to c630ffd Compare December 3, 2021 11:59

tiran force-pushed the bootstrap_again branch from c630ffd to ba35cc4 Compare December 3, 2021 14:23

tiran merged commit 84ca123 into python:main Dec 3, 2021

tiran deleted the bootstrap_again branch December 3, 2021 15:01

bedevere-bot removed the awaiting merge label Dec 3, 2021

tiran mentioned this pull request May 9, 2022

Reintroduce bootstrap_python for freezing #90108

Closed

kumaraditya303 mentioned this pull request Jun 23, 2022

Deepfreeze using 3.10, not bootstrap interpreter. #93771

Closed

corona10 mentioned this pull request Sep 12, 2022

Link-time-optimization with clang is broken #96761

Closed

corona10 mentioned this pull request Oct 11, 2023

test_embed fails if Python is built with LTO and LLVM clang on macOS #110313

Closed

Conversation

tiran commented Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Nov 30, 2021

Uh oh!

bedevere-bot commented Nov 30, 2021

Uh oh!

gvanrossum commented Nov 30, 2021

Uh oh!

vstinner commented Nov 30, 2021

Uh oh!

vstinner commented Nov 30, 2021

Uh oh!

vstinner commented Nov 30, 2021

Uh oh!

tiran commented Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

hroncok commented Dec 2, 2021

Uh oh!

hroncok commented Dec 2, 2021

Uh oh!

tiran commented Dec 2, 2021

Uh oh!

tiran commented Dec 2, 2021

Uh oh!

bedevere-bot commented Dec 2, 2021

Uh oh!

hroncok commented Dec 2, 2021

Uh oh!

tiran commented Dec 3, 2021

Uh oh!

bedevere-bot commented Dec 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tiran commented Nov 30, 2021 •

edited

Loading

tiran commented Dec 1, 2021 •

edited

Loading