[Python-Dev] PEP 393 Summer of Code Project

[Glenn Linderman]

On 8/31/2011 10:20 AM, Guido van Rossum wrote:
> On Wed, Aug 31, 2011 at 1:09 AM, Glenn Linderman<v+python at g.nevcal.com>  wrote:
>> The str type itself can presently be used to process other
>> character encodings: if they are fixed width<  32-bit elements those
>> encodings might be considered Unicode encodings, but there is no requirement
>> that they are, and some operations on str may operate with knowledge of some
>> Unicode semantics, so there are caveats.
> Actually, the str type in Python 3 and the unicode type in Python 2
> are constrained everywhere to either 16-bit or 21-bit "characters".
> (Except when writing C code, which can do any number of invalid things
> so is the equivalent of assuming 1 == 0.) In particular, on a wide
> build, there is no way to get a code point>= 2**21, and I don't want
> PEP 393 to change this. So at best we can use these types to repesent
> arrays of 21-bit unsigned ints. But I think it is more useful to think
> of them as always representing "some form of Unicode", whether that is
> UTF-16 (on narrow builds) or 21-bit code points or perhaps some
> vaguely similar superset -- but for those code units/code points that
> are representable *and* valid (either code points or code units)
> according to the (supported version of) the Unicode standard, the
> meaning of those code points/units matches that of the standard.
>
> Note that this is different from the bytes type, where the meaning of
> a byte is entirely determined by what it means in the programmer's
> head.
>

Sorry, my Perl background is leaking through.  I didn't double check 
that str constrains the values of each element to range 0x110000 but I 
see now by testing that it does.  For some of my ideas, then, either a 
subtype of str would have to be able to relax that constraint, or str 
would not be the appropriate base type to use (but there are other base 
types that could be used, so this is not a serious issue for the ideas).

I have no problem with thinking of str as representing "some form of 
Unicode".  None of my proposals change that, although they may change 
other things, and may invent new forms of Unicode representations. You 
have stated that it is better to document what str actually does, rather 
than attempt to adhere slavishly to Unicode standard concepts.  The 
Unicode Consortium may well define legal, conforming bytestreams for 
communicating processes, but languages and applications are free to use 
other representations internally.  We can either artificially constrain 
ourselves to minor tweaks of the legal conforming bytestreams, or we can 
invent a representation (whether called str or something else) that is 
useful and efficient in practice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110831/2e6d8256/attachment.html>