When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

I merged my PR #9073 instead: it is closer to the current code, allows to implement a new -X coerce_c_locale command line option, and still respects -E and -I option: ignore PYTHONCOERCECLOCALE.

Maybe we can refactor the Python initialization code later, but we need a fix on the current code before 3.7.1 release which is scheduled tomorrow by our release manager, Ned.

What the hell Victor? This is NOT OK!

I've dismissed Victor's review, as he is inappropriately attempting to overrule the PEP process due to his personal preferences.

@vstinner "Closer to the current code" is not an advantage when the structure of the current code goes completely against the way PEP 538 was designed to work (i.e. the CPython runtime never even seeing the original uncoerced locale, except in the locale coercion helper functions)

Initial Travis failure affecting bb16468 was in test_nntplib relating to being unable to assign an address, so I restarted the job.

Got the test_nntplib failure again, so restarted all the CI checks. See https://bugs.python.org/issue19756 (in particular, the last couple of comments)

Moved back to WIP, as I plan to make further changes to the docs wording, and will be making it so that PYTHONCOERCECLOCALE is always handled the same way as the C-level locale configuration variables: read as ASCII independently of the Python -E and -I switches

@vstinner I know you object-in-principle to the idea of having PYTHONCOERCECLOCALE behave the same way as LANG, LC_CTYPE, and LC_ALL do (rather than having it behave like other PYTHON* settings), but I'd still appreciate it if you were willing to review the changes I'm proposing to the CoreConfig initialisation code.

The big items are:

• the coercion related settings go away, replaced with a simple setting related solely to displaying C locale related warnings (or not)
• a separate read_unconditional_env_vars helper function is added

(This isn't urgent, since it's currently only for Python 3.8, and even if a variant of it does get backported, it would be for 3.7.2 at the earliest)

Sorry, I will not review this change. (If you want my opinion, I dislike the PR since it introduces an exception to the -E option and I dislike that.)

Sorry this is taking so long. I wasn't super familiar with the locale stuff so I've been taking my time to make sure I've got it straight. Let me make sure I understand the change correctly.

Status quo:

1. deal with commandline/env (and locale)
   1. init runtime state
   2. default to Py_UTF8Mode
   3. call _Py_SetLocaleFromEnv(LC_CTYPE)
   4. parse commandline
   5. maybe populate core config from env vars (with locale already set), including coerce_c_locale* (from PYTHONCOERCECLOCALE)
   6. call _Py_CoerceLegacyLocale() if coerce_c_locale is set (from step 1.5)...and go back to step 1.4
   7. go back to step 1.4 if switched to UTF8 mode
2. init stdio
3. init core
4. configure main interpreter
5. initialize main interpreter
6. run the program

This PR:

1. call _Py_SetLocaleFromEnv(LC_CTYPE)
2. call Py_CoerceLegacyLocale() if a legacy locale is detected and PYTHONCOERCECLOCALE is set
3. deal with commandline/env (and locale)
   1. init runtime state
   2. default to Py_UTF8Mode
   3. parse commandline
   4. maybe populate core config from env vars, not including coerce_c_locale*
   5. always populate warn_c_locale (using the locale-encoded value from PYTHONCOERCECLOCALE)
   6. go back to step 3 if switched to UTF8 mode
4. init stdio
5. print the locale coercion warning if appropriate
6. init core
7. configure main interpreter
8. initialize main interpreter
9. run the program

Here are the differences I noticed:

Status quo:

• 2 _PyCoreConfig fields (one to indicate that coercion should happen & one for the warning)
• initial locale setting (with LC_CTYPE)
  • right before processing commandline (step 1.3 above)
• reading PYTHONCOERCECLOCALE (step 1.5 above)
  • not used if -E was used
  • relies on currently-set locale (possibly UTF mode)
• _Py_LegacyLocaleDetected()
  • called in config_init_locale() (basically right after env vars are processed; step 1.5 above)
  • used as a fallback if PYTHONCOERCECLOCALE wasn't set
• _Py_CoerceLegacyLocale()
  • called when in the midst of processing the commandline (step 1.6 above)
  • called if PYTHONCOERCECLOCALE set or _Py_LegacyLocaleDetected() returns true
• warning
  • happens immediately when _Py_CoerceLegacyLocale() is called (step 1.6 above)

This PR:

• 1 _PyCoreConfig field (to indicate the warning)
• the locale is initialized to LC_CTYPE immediately (step 1 above)
• reading PYTHONCOERCECLOCALE (step 2 above)
  • used even if -E was used
  • relies only on LC_CTYPE locale
• reading PYTHONCOERCECLOCALE (step 3.5 above)
  • only used if -E was used
  • relies on currently-set locale
• _Py_LegacyLocaleDetected()
  • called right away, before any other locale stuff might have been done or any env vars/cmdline processed (step 2 above)
• _Py_CoerceLegacyLocale()
  • also called right away and only if a legacy locale is detected (step 2 above)
  • called if PYTHONCOERCECLOCALE set (and not "0") and _Py_LegacyLocaleDetected() returns true
• warning
  • happens right before core initialization (step 5 above)

In practice, it seems that the key changes are:

• _Py_CoerceLegacyLocale() is only called if a legacy locale is detected and PYTHONCOERCECLOCALE is set
• -E no longer affects the decision to call _Py_CoerceLegacyLocale()

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

While messing with the current locale setting from a support library rather than the main application still bothers me, I've come to the conclusion that it doesn't bother me enough to want to introduce a non-trivial distinction between the way Python 3.7 handles this scenario, and the way Python 3.8+ handles it.

https://bugs.python.org/issue34589#msg332123 provides a bit more rationale (essentially, I expect that the the combination of circumstances required for there to be an externally observable difference in behaviour between Python 3.7.0 and this PR is going to be sufficiently rare that it simply isn't worth adding a net 200+ lines of code to maintain).

Accordingly, closing this PR and the associated issue indefinitely - while I preferred the original simpler implementation of locale coercion in isolation, all the machinery that the current more complex implementation relies on has to exist to handle UTF-8 mode anyway, so reverting locale coercion back to its original implementation model would end up making the interpreter as a whole more complicated (both to maintain, and to understand).