[IPython-dev] Buffers

[Min RK]

On Jul 26, 2010, at 19:33, Fernando Perez <fperez.net at gmail.com> wrote:

> Glad you worked it out, but I'm worried about one thing: I don't
> believe you can send unicode strings as a buffer over the wire.  The
> reason is that the two interpreters at both ends of the connection
> could have been compiled with different internal unicode encodings.
> Python can be compiled to store unicode internally either as UCS-2 or
> UCS-4, you can check sys.maxunicode to find out how your particular
> build was made:
> 
> http://www.python.org/dev/peps/pep-0100/
> http://www.python.org/dev/peps/pep-0261/
> 
> If you send a unicode string as a buffer from a ucs2 python to a ucs4
> one, you'll get  a mess at the other end, I think.

I'm not sure that is our concern. If buffers are being sent, then it's not zmq whose job it is to interpret that buffer, it's the user's receiving code.

zmq only sends bytes, and for most objects, unicode included, that's what the buffer interface is. But sometimes a unicode object is really just a simple str in a unicode package, and when that's the case we interpret it as a string. Otherwise it's treated like all other objects - a black box that provides a buffer interface. It's up to the user sending the data to send it in a form that they can understand on the other side.

> Minor note:
> On Mon, Jul 26, 2010 at 6:43 PM, Min RK <benjaminrk at gmail.com> wrote:
>> isinstance(s,(str,unicode))
> 
> this is equiv. to: isinstance(s, basestring)
> 

ah, thanks, I hadn't seen that one. I'll use it.

> 
> 
> Cheers,
> 
> f

your points have further clarified that I was mistaken to attempt to support unicode strings. We support basic strings and raw buffers. When faced with a unicode object, we effectively (but not literally) do:
try: send(str(u))
except: send(buffer(u))