Issue 3616: finalizer - Python tracker

If the directory contains invalid filenames (invalid in the system 
charset), an exception is raised by os.path.join() used by 
shutil.rmtree():

    fullname = os.path.join(path, name)
  File "/home/haypo/prog/py3k/Lib/posixpath.py", line 64, in join
    if b.startswith('/'):
TypeError: expected an object with the buffer interface

name is an bytes object, not a str object. My system charset is utf-8.

Example to reproduce the problem:

   mkdir x
   # create a file with an invalid name
   touch -- $(echo -e 'x/--\0250--')
   python -c "import shutils; shutil.rmtree('x')"

=> TypeError: expected an object with the buffer interface

This is certainly a consequence of #3187, which is very high priority
but also very difficult to find a satisying solution to.

Python 2.5 has the same problem (at least, on Linux). rmtree(<unicode 
directory name>) fails if the directory contains invalid unicode 
string. Backtrace:
---
File "shutil.py", line 163, in rmtree
   fullname = os.path.join(path, name)
File "posixpath.py", line 65, in join
   path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xac in position 
3: ordinal not in range(128)
---

The filename:
   $ ls /tmp/run-3/exitcode1/run-4/run-1/session-2/
   ?I?#??????????|?*?  Pum

The instruction was 
rmtree(u"/tmp/run-3/exitcode1/run-4/run-1/session-2").

Selon STINNER Victor <report@bugs.python.org>:
>
> Python 2.5 has the same problem (at least, on Linux). rmtree(<unicode
> directory name>) fails if the directory contains invalid unicode
> string. Backtrace:

Well, I'm not sure we should call it the same problem, although the roots are
the same. The difference is that in 2.x, using bytes strings for file and
directory names is quite normal, especially under Linux where they are just
bytes at the OS level. In 3.0 though, those strings are supposed to be unicode
at the Python level, which makes the problem much more critical.

This issue is marked as closed as a duplicate without a reference to the original task.

I still have this issues on Python 3.4.2, on Windows when shutil.rmtree fails to

Hi, the issue was fixed in Python 3. On Windows, you must use Unicode.
Otherwise, you can get errors like that. On other platforms, Unicode is now
also the best choice on Python 3.

See the second message for the superseder issue.

messages: + msg246888
title: shutil.rmtree() fails on invalid filename -> finalizer