> Martin later added locale.getpreferredencoding(), which tries to
> determine the encoding in a different way new way, based on
> nl_langset(CODEINFO). As you mentioned, this intention was broken
> on several platforms by forcing UTF-8 as output.
When I designed and implemented the PEP 540 (Python UTF-8 Mode), I tried to leave getpreferredencoding() unchanged. The problem was that I quickly got mojibake because too many functions call getpreferredencoding(False):
* open() and _pyio.open() -- in Python 3.10, open() now calls the C _Py_GetLocaleEncoding() function to fix issues during Python shutdown, it also avoids issues at startup.
* Many gettext functions
* cgi to decode the query string from QUERY_STRING env var or sys.argv[1]}
* xml.etree.ElementTree.write(encoding="unicode") is some cases
The Python UTF-8 Mode ignores the locale *on purpose*. But I agree that it's surprising and can lead to confusion. That's what I'm trying to fix here :-)