You should keep things more simple:
- Python and the operation system/filesystem are in a client-server relationship and Python should validate all.
- It doesn't matter what you will finally decide to be the default encoding on various places - all will provide race-conditions with no exception.
- The easiest way to fix this is to give the developer the ability to make a decision (like sys.use_strict_encoding(), sys.setfilesystemencoding(), sys.setdefaultencoding() etc.).
* For example giving the developer control is especially needed if he wants to handle multiple different filesystems.
> Why do you think that the issue is specific to Python 3? Try to open a
> terminal with LC_ALL=C and try to type non-ASCII characters with your
> keyboard. You can't because your terminal uses ASCII.
sworddragon@ubuntu:~$ LANG=C
sworddragon@ubuntu:~$ ä
bash: $'\303\244': command not found
- The terminal doesn't pseudo-crash with an exception because it doesn't matter about encodings.
- It allows to change the encoding at runtime.
> Did you
> applications written in another language handling Unicode, like Perl?
Compare C: It wouldn't matter like the terminal. For example fopen will simply return NULL if it can't open the file 'ä' because the filesystem is endoded with ISO-8859-1 and we wanted to open the utf-8 counterpart.
> Can you explain the "various reasons" why users explictly force the
> encoding to ASCII?
For example I'm using this for testcases to set the language uncomplicated to english.