Do you guys want to chat about this on IRC?<div><br><br><div class="gmail_quote">On Tue, Jul 27, 2010 at 3:16 PM, MinRK <span dir="ltr"><<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Okay, so it sounds like we should never interpret unicode objects as simple strings, if I am understanding the arguments correctly. <div>
<br></div><div>I certainly don't think that sending anything providing the buffer interface should raise an exception, though. It should be up to the user to know whether the buffer will be legible on the other side.</div>
<div><br></div><div>The situation I'm concerned about is that json gives you unicode strings, whether that was the input or not.</div><div>s1 = 'word'</div><div>j = json.dumps(s1)</div><div>s2 = json.loads(j)</div>
<div># u'word'</div><div><br></div><div>Now, if you have that logic internally, and you are sending messages based on messages you received, unless you wrap _every single thing_ you pass to send in str(), then you are calling things like send(u'word'). I really don't think that should not raise an error, and trunk surely does.</div>
<div><br></div><div>The other options are either to always interpret unicode objects like everything else, always sending by its buffer, trusting that the receiving end will call decode (which may require that the message be copied at least one extra time). This would also mean that if A sends something packed by json to B, B unpacks it, and it included a str to be sent to C, then B has a unicode wrapped version of it (not a str). If B then sends it on to C, C will get a string that will _not_ be the same as the one A packed and sent to B. I think this is terrible, since it seems like such an obvious (already done) fix in zmq.</div>
<div><br></div><div>I think that the vast majority of the time you are faced with unicode strings, they are in fact simple str instances that got wrapped, and we should expect that and deal with it.</div><div><br></div><div>
I decided to run some tests, since I currently have a UCS2 (OSX 10.6.4) and UCS4 (ubuntu 10.04) machine</div><div>They are running my `patches' zmq branch right now, and I'm having no problems.</div><div><br></div>
<div>case 1: sys.defaultencoding = utf8 on mac, ascii on ubuntu.</div><div>a.send(u'who') # valid ascii, valid utf-8, ascii string sent</div><div>b.recv()</div><div># 'who'</div><div><br></div><div>u=u'whoπ'</div>
<div># u'who\xcf\x80'</div><div><br></div><div>a.send(u'whoπ') # valid ascii, valid utf-8, utf string sent</div><div>b.recv().decode('utf-8')</div><div># u'who\xcf\x80'</div><div>
<br></div><div>case 2: sys.defaultencoding = ascii,ascii</div><div><div>a.send(u'who') # valid ascii, string sent</div><div>b.recv()</div><div># 'who'</div><div><br></div><div>u=u'whoπ'</div><div>
u</div>
<div># u'who\xcf\x80'</div><div><br></div><div>a.send(u'whoπ') # invalid ascii, buffer sent</div><div>s = b.recv()</div><div># 'w\x00h\x00o\x00\xcf\x00\x80\x00'</div><div>s.decode('utf-8')</div>
<div># UnicodeError (invalid utf-8)</div><div>s.decode('utf16')</div><div># u'who\xcf\x80'</div></div><div><br></div><div><br></div><div>It seems that the _buffer_ of a unicode object is always utf16</div>
<div><br></div><div>I also did it with utf-8 on both sides, and threw in some latin-1, and there was no difference between those and case 1.</div><div><br></div><div>I can't find the problem here.</div><div><br></div>
<div>As far as I can tell, a unicode object is:</div><div>a) a valid string for the sender, and the string is sent in the sender's default encoding</div><div>on the receiver:</div><div> sock.recv().decode(sender.defaultcodec)</div>
<div> gets the object back</div><div>b) not a valid string for the sender, and the utf16 buffer is sent</div><div>on the receiver:</div><div> sock.recv().decode('utf16')</div><div> always seems to work</div>
<div><br></div><div>I even tried various instances of specifying the encoding as latin, etc. and sending math symbols (√,∫) in various directions, and invariably the only thing I needed to know on the receiver was the default encoding on the sender. Everything was reconstructed properly with either s.decode(sender.defaultcodec) or s.decode(utf16), depending solely on whether str(u) would raise on the sender.</div>
<div><br></div><div>Are there specific symbols and/or directions where I should see a problem? Based on reading, I figured that math symbols would if anything, but they certainly don't in either direction.</div><div>
<br>
</div><div>-MinRK</div><div><div></div><div class="h5"><div><br></div><div><br><div class="gmail_quote">On Tue, Jul 27, 2010 at 13:13, Fernando Perez <span dir="ltr"><<a href="http://fperez.net" target="_blank">fperez.net</a>@<a href="http://gmail.com" target="_blank">gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Tue, Jul 27, 2010 at 12:23 PM, Brian Granger <<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a>> wrote:<br>
> This is definitely an issue. Also, someone could set their own custom<br>
> unicode encoding by hand and that would mess this up as well.<br>
><br>
>><br>
>> If it is a problem, then there are some options:<br>
>><br>
>> - disallow communication between ucs 2/4 pythons.<br>
><br>
> But this doesn't account for other encoding/decoding setups.<br>
<br>
</div>Note that when I mention ucs2/4, that refers to the *internal* python<br>
storage of all unicode objects. That is: ucs2/4 is how the buffer,<br>
under the hood for a unicode string, is written in memory. There are<br>
no other encoding/decoding setups for Python, this is strictly a<br>
compile-time flag and can only be either ucs2 or ucs4.<br>
<br>
You can see the value by typing:<br>
<br>
In [1]: sys.maxunicode<br>
Out[1]: 1114111<br>
<br>
That's ucs-4, and that number is the whole of the current unicode<br>
standard. If you get instead 2^16, it means you have a ucs2 build,<br>
and python can only encode strings in the BMP (basic multilingual<br>
plane, where all living languages are stored but not math symbols,<br>
musical symbols and some extended Asian characters).<br>
<br>
Does that make sense?<br>
<br>
Note that additionally, it's exceedingly rare for anyone to set up a<br>
custom encoding for unicode. It's hard to do right, requires plumbing<br>
in the codecs module, and I think Python supports out of the box<br>
enough encodings that I can't imagine why anyone would write a new<br>
encoding. But regardless, if a string has been encoded then it's OK:<br>
now it's bytes, and there's no problem.<br>
<div><br>
>> - detect a mismatch and encode/decode all unicode strings to utf-8 on<br>
>> send/receive, but allow raw buffer sending if there's no mismatch.<br>
><br>
> This will be tough though if users set their own encoding.<br>
<br>
</div>No, the issue with users having something other than utf-8 is<br>
orthogonal to this. The idea would be: - if both ends of the<br>
transmission have conflicting ucs internals, then all unicode strings<br>
are sent as utf-8. If a user sends an encoded string, then that's<br>
just a bunch of bytes and it doesn't matter how they encoded it, since<br>
they will be responsible for decoding it on the other end.<br>
<br>
But I still don't like this approach because the ucs2/4 mismatch is a<br>
pair-wise problem, and for a multi-node setup managing this pair-wise<br>
switching of protocols can be a nightmare. And let's not even get<br>
started on what pub/sub sockets would do with this...<br>
<div><br>
>> - *always* encode/decode.<br>
>><br>
><br>
> I think this is the option that I prefer (having users to this in their<br>
> application code).<br>
<br>
</div>Yes, now that I think of pub/sub sockets, I don't think we have a<br>
choice. It's a bit unfortunate that Python recently decided *not* to<br>
standardize on a storage scheme:<br>
<br>
<a href="http://mail.python.org/pipermail/python-dev/2008-July/080886.html" target="_blank">http://mail.python.org/pipermail/python-dev/2008-July/080886.html</a><br>
<br>
because it means forever paying the price of encoding/decoding in this context.<br>
<br>
Cheers,<br>
<font color="#888888"><br>
f<br>
</font><br>
ps - as you can tell, I've been finally doing my homework on unicode,<br>
in preparation for an eventual 3.x transition :)<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Brian E. Granger, Ph.D.<br>Assistant Professor of Physics<br>Cal Poly State University, San Luis Obispo<br><a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a><br>
<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
</div>