[Python-Dev] PEP 393 Summer of Code Project

[Ezio Melotti]

On Wed, Aug 24, 2011 at 11:37 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/24/2011 1:45 PM, Victor Stinner wrote:
>
>> Le 24/08/2011 02:46, Terry Reedy a écrit :
>>
>
>  I don't think that using UTF-16 with surrogate pairs is really a big
>> problem. A lot of work has been done to hide this. For example,
>> repr(chr(0x10ffff)) now displays '\U0010ffff' instead of two characters.
>> Ezio fixed recently str.is*() methods in Python 3.2+.
>>
>
> I greatly appreciate that he did. The * (lower,upper,title) methods
> apparently are not fixed yet as the corresponding new tests are currently
> skipped for narrow builds.


There are two reasons for this:
1) the str.is* methods get the string and return True/False, so it's enough
to iterate on the string, combine the surrogates, and check if the result
islower/upper/etc.
Methods like lower/upper/etc, afaiu, currently get only a copy of the
string, and modify that in place.  The current macros advance to the next
char during reading and writing, so it's not possible to use them to
read/write from/to the same string.  We could either change the macros to
not advance the pointer [0] (and do it manually in the other functions like
is*) or change the function to get the original string too.
2) I'm on vacation.

Best Regards,
Ezio Melotti

[0]: for lower/upper/title it should be possible to modify the string in
place, because these operations never converts a non-BMP char to a BMP one
(and vice versa), so if two surrogates are read, two surrogates will be
written after the transformation.  I'm not sure this will work with all the
methods though (e.g. str.translate).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110826/89dfd129/attachment.html>