◐ Shell
clean mode source ↗

Message 102996 - Python tracker

Python3 uses unicode surrogates to store undecodable filenames. Eg. the filename b"abc\xff.py" is encoded as "abc\xdcff.py" if the file system encoding is ASCII. Pickle is unable to store them:

./python -c 'import pickle; pickle.dumps("abc\udcff")'
(...)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 20: surrogates not allowed

This is a limitation of pickle (in the binary mode): Python accepts to store any unicode character, but pickle doesn't.

Using "surrogatepass" error handler should be enough to fix this issue.

Related issue: #3672 (Reject surrogates in utf-8 codec) -> r72208 creates "surrogatepass" error handler.