bpo-43112: detect musl as a separate SOABI#24502

ncopa

musl libc and gnu libc are not ABI compatible so we need set different
SOABI for musl and not simply assume that all linux is linux-gnu.

Replace linux-gnu with the detected os for the build from config.guess
for linux-musl*.

https://bugs.python.org/issue43112

Automerge-Triggered-By: GH:brettcannon

musl libc and gnu libc are not ABI compatible so we need set different SOABI for musl and not simply assume that all linux is linux-gnu. Replace linux-gnu with the detected os for the build from config.guess for linux-musl*.

tiran

You also have to re-generate configure script with autoreconf and commit the new configure file.

bedevere-bot

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

ncopa

I have made the requested changes; please review again

bedevere-bot

Thanks for making the requested changes!

@tiran: please review the changes made to this pull request.

tiran

Looks ok to me.

Let's wait for input on https://discuss.python.org/t/pre-pep-platform-tag-for-linux-distributions-using-musl/7165

uranusjr

Unfortunately I don’t have the required autoconf knowledge to understand what’s going on here :(

stratakis

I'd remove the runstatedir changes, as those are generated due to either a later version of autoconf, or from one where that change was backported from upstream. However when someone makes another change to the configure script it can be removed again depending on the autoconf capabilities of the developer's machine.

See also: https://bugs.python.org/issue40216

tiran

Unfortunately I don’t have the required autoconf knowledge to understand what’s going on here :(

The magic happens here:

  case "$build_os" in
  linux-musl*)
    PLATFORM_TRIPLET=`echo "$PLATFORM_TRIPLET" | sed 's/linux-gnu/linux-musl/'`
    ;;
  esac

The shell code roughly translates to

if build_os.startswith("linux-musl"):
    PLATFORM_TRIPLET = PLATFORM_TRIPLET.replace("linux-gnu", "linux-musl")

This changes the SOABI from cpython-39-x86_64-linux-gnu to cpython-39-x86_64-linux-musl on all platforms that identify as Linux with musl libc.

uranusjr

Thanks for the explaination. Where does $build_os come from?

Am I correct to assume this needs to be done in the post-processing instead of conftest.c because musl provides no compile time detection on its usage (instead of glibc)?

tiran

build_os is provided by autoconf, https://www.gnu.org/software/autoconf/manual/autoconf-2.68/autoconf.html#index-build_005fos-1848

We could modify conftest.c, but it would take much more effort. We would have add a dozen ifdef checks for musl. GCC uses a similar approach like this patch in https://github.com/gcc-mirror/gcc/blob/master/gcc/config/t-musl . $(subst ...) is a Makefile command to substitute text.

uranusjr

Thanks. I still can’t say I really know how the code works, but kind of understand now what’s going on, and the approach makes sense to me.

ncopa

I think we may also need something like this:

diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
index e279957e26..83ce74aa6f 100644
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -387,10 +387,13 @@ def test_triplet_in_ext_suffix(self):
         if re.match('(i[3-6]86|x86_64)$', machine):
             if ctypes.sizeof(ctypes.c_char_p()) == 4:
                 self.assertTrue(suffix.endswith('i386-linux-gnu.so') or
-                                suffix.endswith('x86_64-linux-gnux32.so'),
+                                suffix.endswith('x86_64-linux-gnux32.so') or
+                                suffix.endswith('i386-linux-musl.so'),
                                 suffix)
             else: # 8 byte pointer size
-                self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix)
+                self.assertTrue(suffix.endswith('x86_64-linux-gnu.so') or
+                                suffix.endswith('x86_64-linux-musl.so'),
+                                suffix)
 
     @unittest.skipUnless(sys.platform == 'darwin', 'OS X-specific test')
     def test_osx_ext_suffix(self):

github-actions

This PR is stale because it has been open for 30 days with no activity.

alex

@ncopa I believe you are correct that those tests need to be updated

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL8 LTO + PGO 3.x has failed when building commit 1f036ed.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/458/builds/1303) and take a look at the build logs.
Check if the failure is related to this commit (1f036ed) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/458/builds/1303

Failed tests:

test_sysconfig

Failed subtests:

test_triplet_in_ext_suffix - test.test_sysconfig.TestSysConfig

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

418 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 3 sec
test_multiprocessing_spawn: 1 min 47 sec
test_multiprocessing_forkserver: 1 min 14 sec
test_tokenize: 1 min 13 sec
test_multiprocessing_fork: 1 min 8 sec
test_asyncio: 57.7 sec
test_unparse: 52.9 sec
test_capi: 49.0 sec
test_signal: 47.9 sec
test_io: 40.7 sec

1 test failed:
test_sysconfig

13 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

1 re-run test:
test_sysconfig

Total duration: 3 min 58 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_baa5c5d3'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_987e13b0'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/test/test_sysconfig.py", line 439, in test_triplet_in_ext_suffix
    self.assertTrue(suffix.endswith(expected_suffixes),
                                    ^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'expected_suffixes' where it is not associated with a value


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_c047c888'

follow-up to GH-24502

merwok

Thanks tianon, created #31001

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL8 LTO 3.x has failed when building commit 1f036ed.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/361/builds/1306) and take a look at the build logs.
Check if the failure is related to this commit (1f036ed) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/361/builds/1306

Failed tests:

test_sysconfig

Failed subtests:

test_triplet_in_ext_suffix - test.test_sysconfig.TestSysConfig

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

419 tests OK.

10 slowest tests:

test_peg_generator: 7 min 33 sec
test_gdb: 4 min 22 sec
test_concurrent_futures: 3 min 12 sec
test_multiprocessing_spawn: 1 min 58 sec
test_multiprocessing_forkserver: 1 min 18 sec
test_tokenize: 1 min 15 sec
test_multiprocessing_fork: 1 min 8 sec
test_asyncio: 1 min 1 sec
test_unparse: 58.7 sec
test_io: 57.5 sec

1 test failed:
test_sysconfig

12 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_startfile
test_tix test_tk test_ttk_guionly test_winconsoleio test_winreg
test_winsound test_zipfile64

1 re-run test:
test_sysconfig

Total duration: 7 min 56 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/test/test_sysconfig.py", line 439, in test_triplet_in_ext_suffix
    self.assertTrue(suffix.endswith(expected_suffixes),
                                    ^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'expected_suffixes' where it is not associated with a value


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_bce9806a'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_9eb00698'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_104a5c42'

Accidentally introduced by GH-24502

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz> (cherry picked from commit 992fcd1)

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz> (cherry picked from commit 992fcd1)

bpo-43112: detect musl as a separate SOABI …

82bf05d

musl libc and gnu libc are not ABI compatible so we need set different SOABI for musl and not simply assume that all linux is linux-gnu. Replace linux-gnu with the detected os for the build from config.guess for linux-musl*.

the-knights-who-say-ni added the CLA signed label Feb 10, 2021

bedevere-bot added the awaiting review label Feb 10, 2021

📜🤖 Added by blurb_it.

a2f26c7

tiran previously requested changes Feb 10, 2021

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Feb 10, 2021

regenerate configure

e45d9f5

bedevere-bot added awaiting change review and removed awaiting changes labels Feb 10, 2021

bedevere-bot requested a review from tiran February 10, 2021 19:05

Revert the runstate changes in the generated configure

85e0602

github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 2, 2021

tiran removed their request for review April 17, 2021 21:16

lithammer mentioned this pull request Nov 18, 2021

Build musllinux wheels psycopg/psycopg2#1392

Merged

repsejnworb mentioned this pull request Nov 18, 2021

Add image for musllinux_1_2 policy pypa/manylinux#1225

Closed

dvarrazzo mentioned this pull request Nov 23, 2021

Some musllinux binary package not correctly working psycopg/psycopg#161

Closed

merwok added a commit that referenced this pull request Jan 29, 2022

bpo-43112: fix tests …

a331ff9

follow-up to GH-24502

merwok mentioned this pull request Jan 29, 2022

bpo-43112: fix new sysconfig test #31001

Closed

pablogsal pushed a commit that referenced this pull request Jan 29, 2022

Fix a UnboundLocalError bug in test_sysconfig (GH-31000) …

8b1b27f

Accidentally introduced by GH-24502

ncopa deleted the bpo-43112 branch March 16, 2022 10:59

autobakterie reviewed Aug 4, 2022

View reviewed changes

This was referenced Aug 4, 2022

python3: backport and fix target musl libc detection openwrt/packages#19110

Merged

Target libc based suffix (-musl/-gnu) in PLATFORM_TRIPLET is chosen based on build machine configuration instead of configuration of the target #95855

Closed

tlaurion mentioned this pull request Sep 26, 2022

CPython integration help needed linuxboot/heads#689

Open

ncopa mannequin mentioned this pull request Jan 19, 2023

SOABI on Linux does not distinguish between GNU libc and musl libc #87278

Closed

Conversation

ncopa commented Feb 10, 2021 • edited by miss-islington Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tiran left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Feb 10, 2021

Uh oh!

ncopa commented Feb 10, 2021

Uh oh!

bedevere-bot commented Feb 10, 2021

Uh oh!

tiran commented Feb 11, 2021

Uh oh!

uranusjr commented Feb 11, 2021

Uh oh!

stratakis commented Feb 11, 2021

Uh oh!

tiran commented Feb 12, 2021

Uh oh!

uranusjr commented Feb 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tiran commented Feb 12, 2021

Uh oh!

uranusjr commented Feb 12, 2021

Uh oh!

ncopa commented Mar 2, 2021

Uh oh!

github-actions Bot commented Apr 2, 2021

Uh oh!

alex commented Nov 1, 2021

Uh oh!

bedevere-bot commented Jan 29, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

merwok commented Jan 29, 2022

Uh oh!

bedevere-bot commented Jan 29, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

ncopa commented Feb 10, 2021 •

edited by miss-islington

Loading

uranusjr commented Feb 12, 2021 •

edited

Loading