Issue 5410: msvcrt bytes cleanup

I came from issue5391. Here is quote of Victor's message.

>* msvcrt.putch(char), msvcrt.ungetch(char): msvcrt has also:
>  - msvcrt.getch()->byte string of 1 byte
>  - msvcrt.getwch()->unicode string of 1 character
>  - msvcrt.putwch(unicode string of 1 character)
>  - msvcrt_ungetwch(unicode string of 1 character)
>  Hum, putch(), ungetch(), getch() use inconsistent types 
>(unicode/bytes) and should be fixed. Another issue should be open for 
>that.
>
>Notes: msvcrt.putwch() accepts string of length > 1 and 
>msvcrt.ungetwch() doesn't check string length (and so may crash with 
>length=0 or length > 1?).

And msvcrt.ungetwch() calls _ungetch not _ungetwch. Here is the patch
hopefully fixing these issue. (I cannot test wide version of functions
because VC6 don't have them)

> msvcrt.ungetwch() calls _ungetch not _ungetwch

... are you sure that someone already used these functions? :-)

If you suppose that issue5499 is fixed, you can leave msvcrt_putch() 
and msvcrt_ungetch unchanged and use "C" format in msvcrt_ungetwch() 
("Py_UNICODE ch;" have to be replaced by "int ch;" for the 
format "C").

Patch implementing my proposition (depends on issue5499).

issue5499 is fixed, so msvcrt_wchar.patch can now be used :-) Anyone 
available for a review and/or _a test_? I don't have Windows, so it's 
hard for me to test my patch.

There seems to be a problem with ungetwch():

>>> s = msvcrt.getwch()
# Here I type the Euro sign (€)
>>> ascii(s)
"'\\u20ac'"
>>> msvcrt.ungetwch(s)
>>> u = msvcrt.getwch()
>>> ascii(u)
"'\\xac'"

I think this can wait until the first beta.

> There seems to be a problem with ungetwch()

I tested Visual C++ Express 2008 and it looks like _ungetwch() only 
keep 8 lower bits (like _ungetwch(x & 255)). But it's a bug in 
Microsoft library, not in Python code (I added some printf to be 
sure).

My patch (msvcrt_wchar.patch) makes the situation better, but it's not 
perfect because of a bug in Microsoft's library.

msvcrt.getwch() works correctly with characters with code > 255 (eg. 
euro sign, U+20ac, 8364 in decimal).

MSDN says _ungetwch returns WEOF instead of EOF when error occurs.
http://msdn.microsoft.com/en-us/library/yezzac74(VS.80).aspx

I cannot see any remarks about masking behavior. :-(

> I cannot see any remarks about masking behavior. :-(

I asked on a french Windows developer channel. The answer is that the 
Windows terminal uses "ANSI" charset even if it's possible to use 
unicode. So it's a bug in Microsoft msvcrt library (directly in the 
terminal implementation), not in Python.

Anyway I think that my patch (msvcrt_wchar.patch) makes the situation 
better ;-)

> MSDN says _ungetwch returns WEOF instead of EOF when error occurs.

Ok, I updated my patch (to use WEOF).

Applied in r72185.

messages: + msg86125

messages: + msg85410
nosy: + pitrou

messages: + msg83686