Issue 28440: ensurepip and pip install failures on macOS Sierra with non-system Python 2.7.x

Messages (8) msg278649 - (view) Author: Marc Culler (Marc.Culler) * Date: 2016-10-14 13:08

Changes made to /Library/Python on macOSX Sierra cause the --with-ensurepip compiler flag to fail, and lead to failures of pip after installing Python.

The new file that causes the problem on Sierra is:
/Library/Python/2.7/site-packages/Extras.pth
The current version of site.py reads that .pth file, which results in sys.path containing the path:
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
The latter directory (which is protected by SIP) contains many Python 2.7 packages, including easy_install, setuptools, six, py2app, numpy, pylab and pyOpenSSL.  The effect of including this SIP-protected path in sys.path is this:  installing or upgrade a package in /Library/Frameworks/Python.framework which is also installed as an "Extra" in the /System/Frameworks/Python.framework will cause pip to attempt to delete the "old" package from its SIP-protected directory, leading to a "Permission Denied" exception and a failed install.

Given that Apple has now tied /Library/Python to the system Python in this way, thereby making a separate Python framework in /Library/Frameworks unusable, the natural solution to this problem would be to stop including any /Library/Python paths in sys.path.

I am attaching a patch that removes the block of code in site.py which adds these paths.

msg278657 - (view) Author: Ned Deily (ned.deily) * Date: 2016-10-14 16:54

This also affects the -m ensurepip installation of pip itself: the pip install fails trying to upgrade the Apple-supplied version of setuptools.  The behavior of adding the system Python's site-packages directory to the search path of all framework-build Pythons was deliberately added as a result of Issue4865.  I think experience has shown this was not a good idea because of the coupling it introduced between separate Python installations and some third-party distributors of Python on OS X already patch this code out.  It's currently only an issue for 2.7.x since Apple has not yet shipped versions of Python 3.x but the code should be removed there, too.

msg279779 - (view) Author: Ronald Oussoren (ronaldoussoren) * Date: 2016-10-31 11:26

Another reason to remove this feature: installing python 2.7.12 using the installer on www.python.org breaks the system install of Python, likely because of this issue.

I'm a bit sad that this feature has to go, but modern Python use (in particular virtualenv) has reduced the need for this feature.

msg282272 - (view) Author: Roundup Robot (python-dev) Date: 2016-12-03 07:15

New changeset a8a342b3fbc7 by Ned Deily in branch '2.7':
Issue #28440: No longer add /Library/Python/site-packages, the Apple-supplied
https://hg.python.org/cpython/rev/a8a342b3fbc7

msg282275 - (view) Author: Ned Deily (ned.deily) * Date: 2016-12-03 07:21

I've pushed the change to revert Issue4865 for the 2.7 branch for release with 2.7.13.  I am leaving this open to push similar changes for 3.x branches in anticipation of Apple someday supplying system Python 3.x.  It would also be good to see if the post-install script that runs ensurepip can be made to fail more noisily.

msg311261 - (view) Author: Ned Deily (ned.deily) * Date: 2018-01-30 10:14

New changeset 763f094b1f0ce2a07768828e78afe9687e9ad3bb by Ned Deily in branch 'master':
bpo-28440: Don't add /Library/Python/3.x/site-packages to sys.path (#5445)
https://github.com/python/cpython/commit/763f094b1f0ce2a07768828e78afe9687e9ad3bb

msg347833 - (view) Author: Ronald Oussoren (ronaldoussoren) * Date: 2019-07-13 14:45

In response to msg282272: That day has come, the beta for 10.15 contains Python 3.7.3.

msg347840 - (view) Author: Ned Deily (ned.deily) * Date: 2019-07-13 15:23

> That day has come, the beta for 10.15 contains Python 3.7.3.

Yay!  FTR, 763f094b1f0ce2a07768828e78afe9687e9ad3bb was released in 3.7.0.

The only reason I have left this issues open was that I also wanted to make sure that ensurepip failures in the post-install script do not fail silently.

History Date User Action Args 2022-04-11 14:58:38adminsetgithub: 72626 2019-07-13 15:23:04ned.deilysetmessages: + msg347840 2019-07-13 14:45:33ronaldoussorensetmessages: + msg347833 2018-01-30 10:14:14ned.deilysetmessages: + msg311261 2018-01-30 09:52:41ned.deilysetpull_requests: + pull_request5277 2016-12-03 07:21:14ned.deilysetpriority: release blocker ->

messages: + msg282275
versions: + Python 3.6, Python 3.7

2016-12-03 07:15:15python-devsetnosy: + python-dev
messages: + msg282272
2016-10-31 11:26:02ronaldoussorensetmessages: + msg279779 2016-10-14 16:54:39ned.deilysetpriority: normal -> release blocker

type: compile error ->
assignee: ned.deily

title: pip failures on macOS Sierra -> ensurepip and pip install failures on macOS Sierra with non-system Python 2.7.x
nosy: + benjamin.peterson
versions: + Python 2.7
messages: + msg278657
stage: patch review

2016-10-14 15:07:42r.david.murraysetnosy: + ronaldoussoren, ned.deily
components: + macOS
2016-10-14 13:09:00Marc.Cullercreate