Issue 43719: Master build failure on Windows getting file system encoding

After a fresh update from upstream
> PCbuild\build.bat -D (or without -D)
on my machine ends with

  python.vcxproj -> f:\dev\3x\PCbuild\amd64\python_d.pdb (Full PDB)
  Python path configuration:
    PYTHONHOME = (not set)
    PYTHONPATH = 'f:\dev\3x\Lib'
    program name = 'f:\dev\3x\PCbuild\amd64\python_d.exe'
    isolated = 0
    environment = 1
    user site = 1
    import site = 1
    sys._base_executable = 'f:\\dev\\3x\\PCbuild\\amd64\\python_d.exe'
    sys.base_prefix = 'f:\\dev\\3x'
    sys.base_exec_prefix = 'f:\\dev\\3x'
    sys.platlibdir = 'lib'
    sys.executable = 'f:\\dev\\3x\\PCbuild\\amd64\\python_d.exe'
    sys.prefix = 'f:\\dev\\3x'
    sys.exec_prefix = 'f:\\dev\\3x'
    sys.path = [
      'f:\\dev\\3x\\Lib',
      'f:\\dev\\3x\\PCbuild\\amd64\\python310_d.zip',
      'f:\\dev\\3x\\DLLs',
      'f:\\dev\\3x\\lib',
      'f:\\dev\\3x\\PCbuild\\amd64',
    ]
EXEC : Fatal Python warning : init_fs_encoding: failed to get the Python codec
of the filesystem encoding [f:\dev\3x\PCbuild\python.vcxproj]
  Python runtime state: core initialized
  Traceback (most recent call last):
    File "f:\dev\3x\Lib\encodings\__init__.py", line 31, in <module>
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: The command "setlocal
\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: set PYTHONPATH=f:\dev
\3x\Lib\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: "f:\dev\3x\PCbuild\am
d64\python_d.exe" "f:\dev\3x\PC\validate_ucrtbase.py" ucrtbased" exited with co
de -1073741819.
  pythonw.vcxproj -> f:\dev\3x\PCbuild\amd64\pythonw_d.exe
  pythonw.vcxproj -> f:\dev\3x\PCbuild\amd64\pythonw_d.pdb (Full PDB)

EXEC : Fatal Python warning : init_fs_encoding: failed to get the Python codec
of the filesystem encoding [f:\dev\3x\PCbuild\python.vcxproj]
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: The command "setlocal
\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: set PYTHONPATH=f:\dev
\3x\Lib\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: "f:\dev\3x\PCbuild\am
d64\python_d.exe" "f:\dev\3x\PC\validate_ucrtbase.py" ucrtbased" exited with co
de -1073741819.

I believe I pulled and rebuilt fine just a few days ago. Subsequently running
> python
ends with

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
Traceback (most recent call last):
  File "f:\dev\3x\\lib\encodings\__init__.py", line 31, in <module>

Running > chkdsk F: /scan revealed no obvious disk error, but there could be corrupted bytes in some file, but I don't know what file(s) to check (if text) or delete and have git redownload.  Anyone else have a similar problem?

Re ""f:\dev\3x\PC\validate_ucrtbase.py" ucrtbased" exited with co
de -1073741819."

f:\dev\3x>py -3.10 PC\validate_ucrtbase.py ucrtbased
C:\WINDOWS\SYSTEM32\ucrtbased.DLL is version 10.0.14393.33

so the failure is due to running the file with the nonfunctional python_d.exe.

In a text editor, encodings/__init__.py looks fine.
f:\dev\3x>py -3.10 f:\dev\3x\\lib\encodings\__init__.py
(to check text further) fails at line 33
  from . import alias

In any case, line 31 is 'import codecs'.  codecs.py looks fine and
f:\dev\3x>py -3.10 f:\dev\3x\\lib\codecs.py
runs without error, so its text appears to be intact.  Perhaps one of the imports of compiled C fail drastically.

IMO it's https://bugs.python.org/issue27129#msg389991

Terry, it's not clear to me if this is the same issue (at first blush it appears different) but could you see if you have any pyc files in __pycache__ folders within the Tools or Parser directories in your tree that could be from prior builds?  If so, remove them and try a new build?

The build.bat script's clean process is not currently cleaning them up, which can lead to various oddities if you reuse a source tree across the commit Victor references, as that invalidates the older files but they aren't regenerated.

I ran into the same issue on Windows 10 and git bisected it to the same first bad commit, fcb55c0037baab6f98f91ee38ce84b6f874f034a

The issue persists after rm .\Parser\__pycache__\*
My .\Tools directory has no __pycache__.

Dennis, just to make sure I wasn't too literal, some of the __pycache__ folders are in subdirectories (at least in Tools), so just to double check, you did a recursive search beneath Tools and Parser right?

I could not successfully build even with deletion of __pycache__ in subfolders. I finally got the build to succeed after changing the magic number, so I opened GH-25069.

Correction: I opened GH-25172

Thanks everyone for the rapid responses.  The PR fixes the build for me, with a clean test suite pass, so I added reviewers who could merge, a blurb, and my approval.  This can be closed as out-of-date as soon as the (or a) fix is merged.

Fixed by https://github.com/python/cpython/pull/25172

stage: resolved