@methane: Would you mind to have a look at this change?

I'm not sure if it's correct to replace nl_langinfo(CODESET) with _Py_GetLocaleEncodingObjcet() in os.device_encoding(). One effect is that it returns UTF-8 if the Python UTF-8 Mode is enabled. But there is also another effect: on Android and VxWorks, os.device_encoding() now also returns UTF-8 (if the fd is a TTY) rather than nl_langinfo(CODESET).

I'm not sure if it's the same on Android or not?

IMO this change makes os.devide_encoding() and so indirectly open() more consistent with encoding choices in Python.

By the way, I deeply reworked the documentation on encodings, especially the locale encoding the filesystem encoding and error handler (docs.python.org was not updated yet).

@methane @eryksun: I updated my PR to only change the behavior on Unix (non-Windows platforms).

Example showing the impact of this change in practice, on the stdout TTY:

import sys
assert sys.stdout.isatty()
print("stdout encodingg:", sys.stdout.encoding)
reopen_stdout = open(sys.stdout.fileno(), closefd=False)
print("reopen encoding:", reopen_stdout.encoding)
reopen_stdout.close()

Python 3.9 output (old):

stdout encoding: utf-8
reopen encoding: ISO-8859-1

Python 3.10 output (new):

stdout encodingg: utf-8
reopen encoding: UTF-8

IMHO the new behavior is more consistent.

If you want the old behavior, you can explicitly pass encoding=locale.getpreferredencoding(False) to open().

Thanks for the reviews @eryksun and @methane.

@eryksun: The final change leaves Windows unchanged. If you consider that Python has issues on a specific use case, please open a new issue.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO 3.x has failed when building commit 3529718.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/567/builds/936) and take a look at the build logs.
4. Check if the failure is related to this commit (3529718) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/567/builds/936

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

409 tests OK.

10 slowest tests:

• test_concurrent_futures: 2 min 53 sec
• test_peg_generator: 2 min 52 sec
• test_gdb: 1 min 42 sec
• test_multiprocessing_spawn: 1 min 11 sec
• test_multiprocessing_forkserver: 1 min
• test_multiprocessing_fork: 54.2 sec
• test_signal: 47.4 sec
• test_asyncio: 46.2 sec
• test_tokenize: 37.3 sec
• test_io: 35.9 sec

1 test altered the execution environment:
test_asyncio

14 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 5 min 8 sec

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 321, in __del__
    self.close()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 316, in close
    self._ssl_protocol._start_shutdown()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 590, in _start_shutdown
    self._abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 731, in _abort
    self._transport.abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/selector_events.py", line 680, in abort
    self._force_close(None)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed