Confused about XML-RPC Unicode behaviour (examples attached).
Hi all. I've got a bunch of Twisted XML-RPC servers that serve up a wide range of data types include dates. Up until now I've had some hacky code to protect against bad data types being returned to the serialization layer (Nones, unicodes, etc). However, this code turns out to be difficult to maintain, as well as being a modest performance hit. I thought I see if I couldn't clean up some of the code to make it simpler and less hacky, and I found something rather odd: a Twisted XML-RPC server seemingly can't return a xmlrpc.DateTime object it has only just received from the deserialization layer. I've attached a test client, using Twisted, and two test servers, one using Twisted, one using the standard library's SimpleXMLRPCServer. The standard library version succeeds in returning the received xmlrpclib.DateTime unmolested; the Twisted version complains about the message containing unicode: File "/usr/lib/python2.4/site-packages/twisted/internet/abstract.py",line 173, in write raise TypeError("Data must not be unicode") exceptions.TypeError: Data must not be unicode This seems wrong: surely, at the very least, the serialization and deserialization should allow a round-trip unmodified? Put a slightly different way: I don't have a fundamental problem with the restriction that return types not include unicode, but surely in that case, the same system that enforces this should see to it that I don't receive such a type in the first place (given that the original data type I created in the client was a xmlrpclib.DateTime, which is explicitly designed for serialising over XML-RPC!). Are there technical/historical/political reasons for this of which I know nothing? Regards, Ricky
On Wed, 7 Feb 2007 20:41:41 +0200, kgi <iacovou@gmail.com> wrote:
Hi all.
I've got a bunch of Twisted XML-RPC servers that serve up a wide range of data types include dates. Up until now I've had some hacky code to protect against bad data types being returned to the serialization layer (Nones, unicodes, etc). However, this code turns out to be difficult to maintain, as well as being a modest performance hit.
I thought I see if I couldn't clean up some of the code to make it simpler and less hacky, and I found something rather odd: a Twisted XML-RPC server seemingly can't return a xmlrpc.DateTime object it has only just received from the deserialization layer.
I've attached a test client, using Twisted, and two test servers, one using Twisted, one using the standard library's SimpleXMLRPCServer.
The standard library version succeeds in returning the received xmlrpclib.DateTime unmolested; the Twisted version complains about the message containing unicode:
File "/usr/lib/python2.4/site-packages/twisted/internet/abstract.py",line 173, in write raise TypeError("Data must not be unicode") exceptions.TypeError: Data must not be unicode
This seems wrong: surely, at the very least, the serialization and deserialization should allow a round-trip unmodified? Put a slightly different way: I don't have a fundamental problem with the restriction that return types not include unicode, but surely in that case, the same system that enforces this should see to it that I don't receive such a type in the first place (given that the original data type I created in the client was a xmlrpclib.DateTime, which is explicitly designed for serialising over XML-RPC!).
Are there technical/historical/political reasons for this of which I know nothing?
Nope, it's probably just a bug. It looks like it may be a bug in the xmlrpclib module, though. Consider this interaction: exarkun@charm:~$ python Python 2.4.3 (#2, Oct 6 2006, 07:52:30) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import xmlrpclib d1 = xmlrpclib.DateTime() d1 <DateTime '20070207T13:51:36' at -482e1234> s = xmlrpclib.dumps((d1,)) s '<params>\n<param>\n<value><dateTime.iso8601>20070207T13:51:36</dateTime.iso8601></value>\n</param>\n</params>\n' d2 = xmlrpclib.loads(s)[0][0] d2 <DateTime u'20070207T13:51:36' at -482ddab4> xmlrpclib.dumps((d2,)) u'<params>\n<param>\n<value><dateTime.iso8601>20070207T13:51:36</dateTime.iso8601></value>\n</param>\n</params>\n'
Twisted can probably work around the bug here, though. It would be useful if you could file a ticket in the tracker for this (one in Twisted's and one in Python's). Jean-Paul
On Wednesday 07 February 2007 20:53, Jean-Paul Calderone wrote:
Nope, it's probably just a bug. It looks like it may be a bug in the xmlrpclib module, though. Consider this interaction:
Thanks for your reply, JP. You're right that it looks like a bug in xmlrpclib itself; however, it only looks like it's a bug in Python 2.4's version of xmlrpclib; version Python 2.5 has a slightly different xmlrpclib.py that does not exhibit this behaviour; using your example: $ python2.5 Python 2.5 (r25:51908, Oct 6 2006, 15:22:41) [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import xmlrpclib d1 = xmlrpclib.DateTime() d1 <DateTime '20070209T13:02:31' at b7d4896c> s = xmlrpclib.dumps((d1,)) s '<params>\n<param>\n<value><dateTime.iso8601>20070209T13:02:31</dateTime.iso8601></value>\n</param>\n</params>\n' d2 = xmlrpclib.loads(s)[0][0] d2 <DateTime '20070209T13:02:31' at b7d48e2c> xmlrpclib.dumps((d2,)) '<params>\n<param>\n<value><dateTime.iso8601>20070209T13:02:31</dateTime.iso8601></value>\n</param>\n</params>\n'
So obviously someone noticed that 2.4's was buggy from a round-trip point-of-view and corrected it for 2.5. Not sure if that means that they will be unreceptive to patching 2.4. It also looks like one can just drop 2.5's xmlrpclib.py into a 2.4 tree and run compileall.py and the odd behaviour disappears. Having Twisted work around this problem would be a Good Thing, however. I'll file a bug report. Cheers Ricky
On Wednesday 07 February 2007 20:53, Jean-Paul Calderone wrote:
Twisted can probably work around the bug here, though. It would be useful if you could file a ticket in the tracker for this (one in Twisted's and one in Python's).
Ticket filed: http://twistedmatrix.com/trac/ticket/2446 Thanks.
participants (2)
-
Jean-Paul Calderone -
kgi