gh-138171: Migrate iOS testbed location and add Apple build script#138176

freakboy3742

This script simplifies the process of configuring, compiling and packaging an XCframework for an Apple platform. This is an analog of the scripts used to build Android and Emscripten artefacts. It is a precursor to adding GitHub Action builds for iOS, and producing iOS build artefacts.

At present, it only supports iOS, but it has been constructed so that it could be used on any Apple platform (including, potentially, a redistributable macOS XCframework).

The simplest usage of this script is:

  $ python Apple ci iOS

which will:

Clean any pre-existing build artefacts
Configure and make a Python that can be used for the build
Configure and make a Python for each supported iOS architecture and ABI
Combine the outputs of the builds from the previous step into a single
XCframework, merging binaries into a "fat" binary if necessary
Clone a copy of the testbed, configured to use the XCframework
Construct a tarball containing the release artefacts
Run the test suite using the generated XCframework.

This is the complete sequence that would be needed in CI to build and test a candidate release artefact. The output of the command will detect if it's running in a GitHub Actions and use output groups to make the log output easier to digest.

Each individual step can be invoked individually - there are commands to clean, configure-build, make-build, configure-host, make-host, package, and test.

There is also a build command that can be used to combine the configure and make steps for the build Python, an individual host, all hosts, or all builds.

One of the steps is to manage the download of binary artefacts; these artefacts are downloaded to the cross-build/downloads folder by default, but an alternate cache location can be provided.

There are three potentially controversial parts to this PR:

Moving `iOS` to `Apple/iOS`

As part of recognising that iOS is one of (potentially) many Apple platforms, and to prevent future proliferation of top-level directories for other Apple platforms, it migrates the iOS folder into a top level Apple folder. This change isn't strictly required, but if we don't make this change now, there is potential for a proliferation of directories if/when we add support for tvOS, watchOS, visionOS, macCatalyst. It also provides a convenient home for "cross platform" apple concerns, like the testbed script and the build script itself, as well as place for the existing content in the Mac directory to migrate to as part of a cleanup of macOS builds - something that @ned-deily has expressed an interest in doing ¹

Universal simulator binaries

This PR generates a build that includes x86_64 simulator binaries. The iOS simulator slice of the XCframework is effectively a "universal" build covering both x86_64 and ARM64; although it has to be compiled in 2 passes and binaries merged after the fact. It's possible to both compile and test x86_64 binaries on ARM64 machines (pass --simulator "iPhone 16e,arch=x86_64" to the ci or test target, or to the testbed script), although running the test suite in emulator mode currently causes some test failure related to build-details.json not being fully cross-platform aware.

x86_64 isn't a Tier 3 platform for Python due to the difficulties of commissioning a buildbot; but x86_64 is still a supported platform for Apple. If we're going to produce official binaries, it seems to me like we should still support x86_64.

If we choose not to support x86_64, then there will be some changes required to the build script and testbed, as the name of the simulator slice will change from ios_arm64_x86_64_simulator to ios_arm64_simulator. It would also allow removing the most complicated part of this script - the part that does the merging of binaries.

Backporting to 3.13 and 3.14

I've proposed this for backport to 3.13 and 3.14. This is primarily so that the buildbots can be migrated to use this script, and CI jobs can be added for iOS. The directory reorganization is a significant change to make in a backport, especially in the year-old 3.13 release; however, when I floated this idea at the CPython core team summit, @Yhg1s suggested ¹ that while it is a big change, the set of affected iOS users would be small, so the impact wasn't a huge concern.

NOTE: This PR currently contains a duplicate of the iOS/Resources/bin directory; this is needed because the location is hard-coded into the buildbots. Once the location change is confirmed, I can add the new path to the buildbot configuration, and remove the duplicate bin scripts.

Issue: Add mechanism for building iOS XCframeworks #138171

📚 Documentation preview 📚: https://cpython-previews--138176.org.readthedocs.build/

https://pyfound.blogspot.com/2025/06/python-language-summit-2025-python-on-mobile.html ↩ ↩²

…old location.

freakboy3742

The buildbot initially failed because the buildbot configuration references the older iOS/Resource/bin location. As an immediate workaround, I've restored the shims to their old location; if the directory move is accepted as a change, then I can modify the buildbot and remove the duplicate.

freakboy3742

!buildbot iOS

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 68d671b 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138176%2Fmerge

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

iOS ARM64 Simulator PR

AA-Turner

@freakboy3742 I imagine the CODEOWNERS entries will need updating in this PR?

hugovk

+1 to moving these under Apple.

Or should we go a step further and have a build, platform or $bikeshed dir for all of Apple, Android and PC, and whatever comes next (Linux?)

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

freakboy3742

After a discussion with @ambv, I think we don't need to backport this after all. The motivation to backport was so that we buildbots could use the new directory structure and build scripts. This is required because the buildbot can't currently differentiate PR builds based on the underlying Python version.

However, if we fix that underlying problem, we don't need to backport, as we can retain "old style" builds for pre-3.15, and use new-style builds going forward.

On that basis, I'll merge this to main, but not backport, and target adding iOS binaries for 3.15.

freakboy3742

!buildbot iOS

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit ebbbf32 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138176%2Fmerge

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

iOS ARM64 Simulator PR

…cess.

freakboy3742

!buildbot iOS

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit e0f6b5a 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138176%2Fmerge

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

iOS ARM64 Simulator PR

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86 Debian Non-Debug with X 3.x (no tier) has failed when building commit 35c7e52.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1245/builds/6433) and take a look at the build logs.
Check if the failure is related to this commit (35c7e52) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1245/builds/6433

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/contextlib.py"�[0m, line �[35m85�[0m, in �[35minner�[0m
    return func(*args, **kwds)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m596�[0m, in �[35mtest_interrupt�[0m
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/contextlib.py"�[0m, line �[35m85�[0m, in �[35minner�[0m
    return func(*args, **kwds)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m577�[0m, in �[35m_kill_process�[0m
    self.assertEqual(�[31mjoin�[0m�[1;31m()�[0m, None)
                     �[31m~~~~�[0m�[1;31m^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m250�[0m, in �[35m__call__�[0m
    return �[31mself.func�[0m�[1;31m(*args, **kwds)�[0m
           �[31m~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/multiprocessing/process.py"�[0m, line �[35m156�[0m, in �[35mjoin�[0m
    res = self._popen.wait(timeout)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/multiprocessing/popen_fork.py"�[0m, line �[35m44�[0m, in �[35mwait�[0m
    return �[31mself.poll�[0m�[1;31m(os.WNOHANG if timeout == 0.0 else 0)�[0m
           �[31m~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/multiprocessing/popen_fork.py"�[0m, line �[35m28�[0m, in �[35mpoll�[0m
    pid, sts = �[31mos.waitpid�[0m�[1;31m(self.pid, flag)�[0m
               �[31m~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m573�[0m, in �[35mhandler�[0m
    raise RuntimeError('join took too long: %s' % p)
�[1;35mRuntimeError�[0m: �[35mjoin took too long: <Process name='Process-1' pid=28784 parent=28782 started daemon>�[0m


Traceback (most recent call last):
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/contextlib.py"�[0m, line �[35m85�[0m, in �[35minner�[0m
    return func(*args, **kwds)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m596�[0m, in �[35mtest_interrupt�[0m
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/contextlib.py"�[0m, line �[35m85�[0m, in �[35minner�[0m
    return func(*args, **kwds)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m577�[0m, in �[35m_kill_process�[0m
    self.assertEqual(�[31mjoin�[0m�[1;31m()�[0m, None)
                     �[31m~~~~�[0m�[1;31m^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m250�[0m, in �[35m__call__�[0m
    return �[31mself.func�[0m�[1;31m(*args, **kwds)�[0m
           �[31m~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/multiprocessing/process.py"�[0m, line �[35m156�[0m, in �[35mjoin�[0m
    res = self._popen.wait(timeout)
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/multiprocessing/popen_fork.py"�[0m, line �[35m44�[0m, in �[35mwait�[0m
    return �[31mself.poll�[0m�[1;31m(os.WNOHANG if timeout == 0.0 else 0)�[0m
           �[31m~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/multiprocessing/popen_fork.py"�[0m, line �[35m28�[0m, in �[35mpoll�[0m
    pid, sts = �[31mos.waitpid�[0m�[1;31m(self.pid, flag)�[0m
               �[31m~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-debian-x86.nondebug/build/Lib/test/_test_multiprocessing.py"�[0m, line �[35m573�[0m, in �[35mhandler�[0m
    raise RuntimeError('join took too long: %s' % p)
�[1;35mRuntimeError�[0m: �[35mjoin took too long: <Process name='Process-158' pid=21525 parent=20497 started daemon>�[0m

miss-islington-app

Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington-app

Sorry, @freakboy3742, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 35c7e52b3ea3fdeb8eb77d2d8c803467a2ba6311 3.14

freakboy3742

After a conversation with @hugovk, it might be worth back porting this to 3.14 so that it is in 3.14 final; even if we don't do a binary release until a 3.14.1 or later - or at all - having the directory renamed will make other maintenance and back porting easier.

…ipt (python#138176) Adds tooling to generate and test an iOS XCframework, in a way that will also facilitate adding other XCframework targets for other Apple platforms (tvOS, watchOS, visionOS and even macOS, potentially). --------- Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> (cherry picked from commit 35c7e52)

bedevere-app

GH-139204 is a backport of this pull request to the 3.14 branch.

…ipt (python#138176) Adds tooling to generate and test an iOS XCframework, in a way that will also facilitate adding other XCframework targets for other Apple platforms (tvOS, watchOS, visionOS and even macOS, potentially). --------- Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

…ript (#138176) (#139204) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

…ild script (python#138176) Adds tooling to generate and test an iOS XCframework, in a way that will also facilitate adding other XCframework targets for other Apple platforms (tvOS, watchOS, visionOS and even macOS, potentially). --------- Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Migrate testbed location and add cross-platform build script.

4a11615

freakboy3742 requested review from a team, AA-Turner, corona10 and erlend-aasland as code owners August 27, 2025 06:59

freakboy3742 added OS-ios needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Aug 27, 2025

bedevere-app Bot added the awaiting core review label Aug 27, 2025

bedevere-app Bot mentioned this pull request Aug 27, 2025

Add mechanism for building iOS XCframeworks #138171

Closed

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

As a buildbot workaround, retain a copy of the compiler shims in the … …

b89f46f

…old location.

This comment was marked as outdated.

Sign in to view

...and get the legacy location correct.

68d671b

AA-Turner reviewed Aug 27, 2025

View reviewed changes

freakboy3742 requested a review from ned-deily August 27, 2025 22:09

Updates to CODEOWNERS, gitignore and clean targets.

39856d1

freakboy3742 requested review from ezio-melotti and hugovk as code owners August 27, 2025 22:21

hugovk reviewed Aug 28, 2025

View reviewed changes

freakboy3742 and others added 3 commits August 29, 2025 07:25

Apply suggestions from code review …

f6a836c

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Convert README to Markdown.

c692e9a

Add type hints and more docstrings.

fe39aa4

freakboy3742 removed the needs backport to 3.14 bugs and security fixes label Sep 19, 2025

Move build script into stub XCframework so that 'make testios' has ac… …

e0f6b5a

…cess.

mhsmith reviewed Sep 19, 2025

View reviewed changes

freakboy3742 deleted the apple-build branch September 19, 2025 12:23

bedevere-app Bot removed the awaiting merge label Sep 19, 2025

freakboy3742 added the needs backport to 3.14 bugs and security fixes label Sep 21, 2025

miss-islington-app Bot assigned freakboy3742 Sep 21, 2025

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 21, 2025

hugovk added a commit that referenced this pull request Sep 29, 2025

[3.14] gh-138171: Migrate iOS testbed location and add Apple build sc… …

91d8645

…ript (#138176) (#139204) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Conversation

freakboy3742 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Moving iOS to Apple/iOS

Universal simulator binaries

Backporting to 3.13 and 3.14

Footnotes

Uh oh!

This comment was marked as resolved.

This comment was marked as outdated.

freakboy3742 commented Aug 27, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

freakboy3742 commented Aug 27, 2025

Uh oh!

bedevere-bot commented Aug 27, 2025

Uh oh!

AA-Turner commented Aug 27, 2025

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

freakboy3742 commented Sep 19, 2025

Uh oh!

freakboy3742 commented Sep 19, 2025

Uh oh!

bedevere-bot commented Sep 19, 2025

Uh oh!

freakboy3742 commented Sep 19, 2025

Uh oh!

bedevere-bot commented Sep 19, 2025

Uh oh!

Uh oh!

bedevere-bot commented Sep 19, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

miss-islington-app Bot commented Sep 21, 2025

Uh oh!

miss-islington-app Bot commented Sep 21, 2025

Uh oh!

freakboy3742 commented Sep 21, 2025

Uh oh!

bedevere-app Bot commented Sep 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

freakboy3742 commented Aug 27, 2025 •

edited

Loading

Moving `iOS` to `Apple/iOS`