[Python-Dev] PEP 528: Change Windows console encoding to UTF-8

Steve Dower steve.dower at python.org
Mon Sep 5 15:30:51 EDT 2016


On 05Sep2016 1110, Paul Moore wrote:
> On 5 September 2016 at 18:38, Steve Dower <steve.dower at python.org> wrote:
>>> Can you provide an example of how I'd rewrite the code that I quoted
>>> previously to follow this advice? Note - this is not theoretical, I
>>> expect to have to provide a PR to fix exactly this code should this
>>> change go in. At the moment I can't find a way that doesn't impact the
>>> (currently working and not expected to need any change) Unix version
>>> of the code, most likely I'll have to add buffering of 4-byte reads
>>> (which as you say is complex).
>>
>> The easiest way to follow it is to use "sys.stdin.buffer.read(1)" rather
>> than "sys.stdin.buffer.raw.read(1)".
>
> I may have got confused here. If I say sys.stdin.buffer.read(1),
> having first checked via kbhit() that there's a character
> available[1], then I will always get 1 byte returned, never the
> "buffer too small to return a full character" error that you talk
> about in the PEP? If so, then I don't understand when the error you
> propose will be raised (unless your comment here is based on what you
> say below that we'll now buffer and therefore the error is no longer
> needed).

I don't think using buffer.read and kbhit together is going to be 
reliable anyway, as you may not have read everything that's already 
buffered yet. It's likely feasible if you flush everything, but 
otherwise it's a bit messy.

> One thing I did think of, though - if someone *is* working at the raw
> IO level, they have to be prepared for the new "buffer too small to
> return a full character" error. That's OK. But what if they request
> reading 7 bytes, but the input consists of 6 character s that encode
> to 1 byte in UTF-8, followed by a character that encodes to 2 bytes?
> You can return 6 bytes, that's fine - but you'll presumably still need
> to read the extra character before you can determine that it won't fit
> - so you're still going to have to buffer to some degree, surely? I
> guess this is implementation details, though - I'll try to find some
> time to read the patch in order to understand this. It's not something
> that matters in terms of the PEP anyway, it's an implementation
> detail.

If you do raw.read(7), we internally do "7 / 4" and decide to only read 
one wchar_t from the console. So the returned bytes will be between 1 
and 4 bytes long and there will be more info waiting for next time you ask.

The only case we can reasonably handle at the raw layer is "n / 4" is 
zero but n != 0, in which case we can read and cache up to 4 bytes (one 
wchar_t) and then return those in future calls. If we try to cache any 
more than that we're substituting for buffered reader, which I don't 
want to do.

Does caching up to one (Unicode) character at a time sound reasonable? I 
think that won't be much trouble, since there's no interference between 
system calls in that case and it will be consistent with POSIX behaviour.

Cheers,
Steve



More information about the Python-Dev mailing list