◐ Shell
reader mode source ↗
Skip to content

GH-80789: Bundle ensurepip wheels at build time#109130

Closed
AA-Turner wants to merge 12 commits into
python:mainfrom
AA-Turner:ensurepip/bundler-tools
Closed

GH-80789: Bundle ensurepip wheels at build time#109130
AA-Turner wants to merge 12 commits into
python:mainfrom
AA-Turner:ensurepip/bundler-tools

Conversation

@AA-Turner

@AA-Turner AA-Turner commented Sep 8, 2023

Copy link
Copy Markdown
Member

Based on the discussion in #12791, this is a sketch of a different approach (building very heavily on @webknjaz's work).

We add a fairly simple bundler script at Tools/build/bundle_ensurepip_wheels.py, with two changes to ensurepip itself to adapt. This PR doesn't attempt to switch to the one-project model, as I found the diff was too large to reasonably review.

One question: should we make this part of the default build process (i.e. integration into make all / PCBuild)?

A


📚 Documentation preview 📚: https://cpython-previews--109130.org.readthedocs.build/

webknjaz and others added 4 commits September 7, 2023 00:32
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
@AA-Turner AA-Turner requested a review from a team as a code owner September 8, 2023 10:56
@AA-Turner

Copy link
Copy Markdown
Member Author

0aaf135 was an attempt to integrate this into the Makefile/PCBuild more 'properly', but I'm not confident in making wider-scale changes here (said commit didn't work, so I've reverted it).

@zware zware left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

This is going to need something for buildbots to avoid breaking them all, but I like the direction here.

@ned-deily

Copy link
Copy Markdown
Member

Adding @ambv and @Yhg1s for review of impact on release process and supply chain security.

Reviewed-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
@webknjaz

Copy link
Copy Markdown
Member

This is going to need something for buildbots to avoid breaking them all, but I like the direction here.

I identified the place needing to be changed in #12791 (comment).

@AA-Turner FYI

@AA-Turner

Copy link
Copy Markdown
Member Author

I identified the place needing to be changed

I tried to explore a different approach of fully integrating this into make all in 0aaf135, which I think may be a better approach as it doesn't require downsteam distributors to change anything in their processes. I defer to those with much more experience with the build systems, though.

A

@webknjaz

Copy link
Copy Markdown
Member

I tried to explore a different approach of fully integrating this into make all

I tried doing that in my PR and was getting weird behavior, I wasn't sure why. Maybe I was misusing something in the build machinery or was just confused...

If that works, it's probably fine. OTOH, you may want to take into account that many distributions prefer their build envs to be disconnected from the internet so it might still be wise to provide them with means to pre-provision the wheel separately.

Tagging @encukou @hroncok @mgorny for downstream opinions.

@webknjaz

Copy link
Copy Markdown
Member

This PR doesn't attempt to switch to the one-project model, as I found the diff was too large to reasonably review.

Would #109245 help?

@mgorny

mgorny commented Sep 11, 2023

Copy link
Copy Markdown
Contributor

If that works, it's probably fine. OTOH, you may want to take into account that many distributions prefer their build envs to be disconnected from the internet so it might still be wise to provide them with means to pre-provision the wheel separately.

In Gentoo we aren't actually using the wheels originally bundled with CPython itself but providing the newest versions of the them separately, so as long as that continues working, I suppose that's fine with us. However, it is is paramount that:

  1. The build process and test suite fully respect --with-wheel-pkg-dir and do not attempt to download anything.
  2. This is entirely restricted to CPython build-time, and in particularly creating venv doesn't attempt any fetching.

However, I can imagine it could be mildly annoying to users who aren't used to the new workflow that having a complete git clone would no longer suffice to actually build CPython offline.

@hroncok

hroncok commented Sep 11, 2023

Copy link
Copy Markdown
Contributor

Same for Fedora. With an addition that we do sometimes keep the bundled wheels for too new or too old Pythons, so having a way to pre-populate the wheels instead of downloaidng them would still be needed.

@pradyunsg

Copy link
Copy Markdown
Member

@AA-Turner could you resolve the conflicts with this PR?

I think it would be good to have this be in Python 3.13's cycle as early as possible, to allow any potential redistributors who care about the details of ensurepip to have time to adjust things as well as provide us feedback on this as early as feasible.

@pradyunsg

pradyunsg commented Oct 12, 2023

Copy link
Copy Markdown
Member

Closing this per #80789 (comment). Thanks @AA-Turner for filing this, even though we're not going ahead with this (despite me implying that earlier in the discussion).

@pradyunsg pradyunsg closed this Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants