bpo-9949: Enable symlink traversal for ntpath.realpath#15287

zooba

https://bugs.python.org/issue9949

zooba

This is ready to merge once CI passes. I'll do it and then rebase #15231 before merging that one.

miss-islington

Thanks @zooba for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

(cherry picked from commit 75e0649) Co-authored-by: Steve Dower <steve.dower@python.org>

bedevere-bot

GH-15367 is a backport of this pull request to the 3.8 branch.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows7 SP1 3.x has failed when building commit 75e0649.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/40/builds/2885) and take a look at the build logs.
Check if the failure is related to this commit (75e0649) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/40/builds/2885

Failed tests:

test_venv
test_ntpath

Failed subtests:

test_realpath_curdir - test.test_ntpath.TestNtpath
test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

384 tests OK.

10 slowest tests:

test_multiprocessing_spawn: 2 min 41 sec
test_tools: 2 min 19 sec
test_concurrent_futures: 1 min 37 sec
test_tokenize: 1 min 33 sec
test_lib2to3: 1 min 13 sec
test_asyncio: 1 min 5 sec
test_io: 1 min 3 sec
test_mmap: 1 min 3 sec
test_socket: 58 sec 471 ms
test_venv: 54 sec 83 ms

2 tests failed:
test_ntpath test_venv

33 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_tix test_tk test_ttk_guionly
test_wait3 test_wait4 test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 11 min 4 sec

Click to see traceback logs

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmprmrzilsg\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmp9l9odtp0\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 75e0649.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/3/builds/3298) and take a look at the build logs.
Check if the failure is related to this commit (75e0649) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/3/builds/3298

Failed tests:

test_venv
test_ntpath

Failed subtests:

test_realpath_curdir - test.test_ntpath.TestNtpath
test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

387 tests OK.

10 slowest tests:

test_mmap: 5 min 51 sec
test_tools: 3 min 8 sec
test_multiprocessing_spawn: 2 min 59 sec
test_io: 2 min 39 sec
test_largefile: 2 min 25 sec
test_tokenize: 2 min 19 sec
test_lib2to3: 1 min 55 sec
test_concurrent_futures: 1 min 43 sec
test_pickle: 55 sec 907 ms
test_asyncio: 55 sec 177 ms

2 tests failed:
test_ntpath test_venv

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 13 min 37 sec

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmpjvhg3gb8\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmp6vl8d2pf\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

(cherry picked from commit 75e0649) Co-authored-by: Steve Dower <steve.dower@python.org>

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows7 SP1 3.8 has failed when building commit c30c869.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/208/builds/279) and take a look at the build logs.
Check if the failure is related to this commit (c30c869) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/208/builds/279

Failed tests:

test_venv
test_ntpath

Failed subtests:

test_realpath_curdir - test.test_ntpath.TestNtpath
test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

388 tests OK.

10 slowest tests:

test_multiprocessing_spawn: 2 min 58 sec
test_tools: 2 min 47 sec
test_concurrent_futures: 1 min 49 sec
test_tokenize: 1 min 34 sec
test_zipfile: 1 min 14 sec
test_lib2to3: 1 min 13 sec
test_mmap: 1 min 9 sec
test_asyncio: 1 min 3 sec
test_socket: 57 sec 298 ms
test_venv: 56 sec 438 ms

2 tests failed:
test_ntpath test_venv

33 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_tix test_tk test_ttk_guionly
test_wait3 test_wait4 test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 11 min 14 sec

Click to see traceback logs

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmpyua593m9\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmpux9eusss\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.8 has failed when building commit c30c869.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/217/builds/269) and take a look at the build logs.
Check if the failure is related to this commit (c30c869) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/217/builds/269

Failed tests:

test_venv
test_ntpath

Failed subtests:

test_realpath_curdir - test.test_ntpath.TestNtpath
test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

391 tests OK.

10 slowest tests:

test_mmap: 5 min 49 sec
test_io: 2 min 47 sec
test_multiprocessing_spawn: 2 min 38 sec
test_largefile: 2 min 26 sec
test_tools: 2 min 10 sec
test_tokenize: 1 min 44 sec
test_concurrent_futures: 1 min 41 sec
test_lib2to3: 1 min 21 sec
test_asyncio: 57 sec 360 ms
test_pickle: 46 sec 703 ms

2 tests failed:
test_ntpath test_venv

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 14 min 45 sec

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmp_ejyuk4y\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmp7utx1za6\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86 Windows7 3.8 has failed when building commit c30c869.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/210/builds/208) and take a look at the build logs.
Check if the failure is related to this commit (c30c869) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/210/builds/208

Failed tests:

test_venv
test_ntpath

Failed subtests:

test_realpath_curdir - test.test_ntpath.TestNtpath
test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

391 tests OK.

10 slowest tests:

test_multiprocessing_spawn: 15 min 18 sec
test_zipfile: 7 min 48 sec
test_concurrent_futures: 4 min 44 sec
test_tarfile: 3 min 18 sec
test_pickle: 3 min 15 sec
test_lzma: 3 min 2 sec
test_asyncio: 3 min 372 ms
test_capi: 2 min 57 sec
test_regrtest: 2 min 52 sec
test_venv: 2 min 39 sec

2 tests failed:
test_ntpath test_venv

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 1 hour 5 min

Click to see traceback logs

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmpddif0hoz\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmpdfbt5vu7\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \

[WIP] bpo-9949: Enable symlink traversal for ntpath.realpath

b65f06f

the-knights-who-say-ni added the CLA signed label Aug 14, 2019

bedevere-bot added the awaiting core review label Aug 14, 2019

Fixes link resolution and test file creation

df54868

zooba marked this pull request as ready for review August 14, 2019 20:42

zooba added the needs backport to 3.8 label Aug 14, 2019

zooba added 6 commits August 14, 2019 14:20

Fix x-plat handling in test_ntpath and additional error code

f82129b

Fix use of os.path in ntpath tests

8ee4046

Fix additional uses of os.path

b93c9e8

Validate that the path prefix is unnecessary before removing

5f9816f

Don't need second call to _getfinalpathname

d1d0726

Fix test discovery mocks

9bc4192

zooba mentioned this pull request Aug 15, 2019

[DO NOT MERGE] bugs 9949 and 37834 together #15310

Closed

zooba added 4 commits August 15, 2019 16:13

Avoid traversing a symlink cycle

9dc0608

Simplify and document cycle handling in os.path.realpath

28c9bb4

Update whatsnew

ab485a5

Add fast resolution path and minor doc update

5d560b8

zooba merged commit 75e0649 into python:master Aug 21, 2019

zooba deleted the bpo-9949 branch August 21, 2019 20:43

bedevere-bot removed the awaiting core review label Aug 21, 2019

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 21, 2019

bpo-9949: Enable symlink traversal for ntpath.realpath (pythonGH-15287) …

1813896

(cherry picked from commit 75e0649) Co-authored-by: Steve Dower <steve.dower@python.org>

bedevere-bot removed the needs backport to 3.8 label Aug 21, 2019

miss-islington added a commit that referenced this pull request Aug 21, 2019

bpo-9949: Enable symlink traversal for ntpath.realpath (GH-15287) …

c30c869

(cherry picked from commit 75e0649) Co-authored-by: Steve Dower <steve.dower@python.org>

eryksun reviewed Aug 22, 2019

View reviewed changes

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-9949: Enable symlink traversal for ntpath.realpath (pythonGH-15287)

f06ba72

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-9949: Enable symlink traversal for ntpath.realpath (pythonGH-15287)

03dceb4

websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020

bpo-9949: Enable symlink traversal for ntpath.realpath (pythonGH-15287)

668a94e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-9949: Enable symlink traversal for ntpath.realpath#15287