[Python-Dev] The C API and wide unicode support

[M.-A. Lemburg]

Michael Hudson wrote:
> Guido van Rossum <guido@python.org> writes:
> 
> 
>>OTOH, Michael, is this really something you cannot live with?  Or is
>>it simply a surprise?
> 
> 
> Here's where the problem came up.
> 
> A user posted to pygame-users saying that when he tried to import
> pygame.event, along the lines of PyUnicodeUCS2_Unicode undefined.
> This obviously made a light go on in my head, and I asked where he'd
> got his Python and his pygame.  He'd got his Python from the Redhat
> 7.3 RPM and his pygame from pygame.org.  I suggested building pygame
> from source, which he did and everything worked[*].  
> 
> Prediction: this is going to cause pain.  For instance, if this user
> decides that he wants to upgrade to 2.2.1, he might download Sean's
> RPMs from python.org which are narrow unicode builds -- and then his
> extensions will break.  The problem here is that the kind of users
> this is going to trouble are exactly the users who will not know
> what's going on.

It's a pain, yes, but still better than having seg faults
due to memory corruption afterwords.

> We can't prevent this sort of thing totally, but I think it should be
> possible to carry out simple unicode manipulations (like this example
> of returning a buffer) without incurring this kind of binary
> compatibility worry.  Maybe a "safe" api, plastered with warning signs
> in the docs about poking into the internal structure of the objects.

Perhaps we need an additional abstract API PyObject_UnicodeEx()
which provides a way to additionally define the encoding to assume
for decoding string objects ? (PyObject_Unicode() always assumes
the default encoding)

> I wonder why Redhat distribute wide unicode builds?  That's the
> immediate cause of the problem.  Maybe we could ask them...
> 
> Cheers,
> M.
> [*] actually, I think pygame might break with a wide unicode build.

Why's that ?

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/