@Haypo, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zooba, @benjaminp and @serhiy-storchaka to be potential reviewers.

Ping @ncoghlan - more changes to startup sequence :)

FYI I didn't look again at my code. I only recreated this pull request to not loose the code, since my old PR was on the old CPython repository.

I still have to update the PEP and post it to python-dev.

Questions like "wcsicmp? Any need to be case-sensitive here?" should be asked on the PEP directly, not on the implementation.

You should wait until the PEP is accepted before reviewing the implementation. The PEP may still change.

@Haypo Ah fair enough :) I haven't been paying attention to all the email recently, so I figured it was accepted already.

The implementation is not complete: the command line arguments and environment variables are not re-decoded from UTF-8 once Py_Main() detects that the user requested the UTF-8 mode (-X utf8 or PYTHONUTF8). But thanks to my refactoring work on Modules/main.c ( bugs.python.org/issue32030 ), fixing this issue should now be much simpler.

I updated the PR for the 4th version of the PEP 540: getpreferredencoding() now returns UTF-8 in the UTF-8 Mode.

@methane: You approved my PEP. This is implementation, all tests pass on Linux and Windows. While the implementation is not perfect, I propose to merge it right now to be get more time before the Python 3.7 final to test it.

Remaining issue: when the UTF-8 mode is enabled manually (-X utf8), the command line arguments and environment variables are decoded from the locale encoding instead of being decoded from UTF-8. The fix should be easy, but I didn't have time yet to implement it.

Another less important issue: the code to check if the LC_CTYPE locale is the POSIX locale is not currently shared between locale coercion (PEP 538) and UTF-8 Mode (PEP 540).

I feel adding more code to Program/python.c is bad smell.
Py_Main() is very high level API, but very hard to use in right way for Unix users.
How about adding Py_UnixMain() and move all code in main() to it?

And when -X utf8 option is found, we can decode from char **argv again.
Since mbstowcs() doesn't guarantee round tripping, it is better than re-encode
wchar_t **argv.

I feel adding more code to Program/python.c is bad smell.

It is.

Py_Main() is very high level API, but very hard to use in right way for Unix users. How about adding Py_UnixMain() and move all code in main() to it?

main() and Py_Main() are very complex. With the PEP 432, Nick Coghlan, Eric Snow and me are working on making this code better. See for example https://bugs.python.org/issue32030

Currently, Py_Main() (Modules/main.c) and _PyPathConfig_Calculate() (Modules/getpath.c and PC/getpathp.c) are fully implemented with wchar_t*.

Parsing the command line options using char* on Unix but wchar_t* would make the code more complex. I expect that many lines of code would have to be duplicate, one version for char*, one version for wchar_t*.

For all these reasons, I propose to merge this uncomplete PR and write a different PR for the most complex part, re-encode wchar_t* command line arguments, implement Py_UnixMain() or another even better option?

Oops, I forgot to push my main.c change. It's now done.