Hello, I am trying to use xml-rpc to be able to run some simulations remotely. I am running into a problem with the transfer of numpy arrays. my server code looks like: #!/usr/bin/env python def again(x): # test out the sending of data return [x,x] from SimpleXMLRPCServer import SimpleXMLRPCServer SimpleXMLRPCServer.allow_reuse_address = 1 server = SimpleXMLRPCServer(("", 8000)) server.register_function(again) try: print "Serving..." server.serve_forever() # Start the server finally: print "done." server.server_close() my client code looks like: import numpy from xmlrpclib import ServerProxy server=ServerProxy('http://localhost:8000') server.again(5) # this works b=numpy.random.rand(5,5) server.again(b) # this gives an error this gives the error: <type 'exceptions.TypeError'>: cannot marshal <type 'numpy.ndarray'> objects which seems to be a deficiency of the marshal library, or perhaps I am doing something wrong. Is there a way to fix this? Is there another approach that I should be using? thanks, Brian Blais -- Brian Blais bblais@bryant.edu http://web.bryant.edu/~bblais
Hi, in order to marshal numpy arrays, you can use the tostring() method. The inverse is the fromstring() function in numpy. But you must know dtype and shape in order to reconstruct your array. Greetings, Uwe On 30 Sep., 19:53, Brian Blais <bbl...@bryant.edu> wrote:
Hello,
I am trying to use xml-rpc to be able to run some simulations remotely. I am running into a problem with the transfer of numpy arrays. my server code looks like:
#!/usr/bin/env python def again(x): # test out the sending of data return [x,x]
from SimpleXMLRPCServer import SimpleXMLRPCServer SimpleXMLRPCServer.allow_reuse_address = 1
server = SimpleXMLRPCServer(("", 8000))
server.register_function(again) try: print "Serving..." server.serve_forever() # Start the server finally: print "done." server.server_close()
my client code looks like:
import numpy from xmlrpclib import ServerProxy server=ServerProxy('http://localhost:8000') server.again(5) # this works
b=numpy.random.rand(5,5)
server.again(b) # this gives an error
this gives the error: <type 'exceptions.TypeError'>: cannot marshal <type 'numpy.ndarray'> objects
which seems to be a deficiency of the marshal library, or perhaps I am doing something wrong. Is there a way to fix this? Is there another approach that I should be using?
thanks,
Brian Blais
-- Brian Blais bbl...@bryant.eduhttp://web.bryant.edu/~bblais
_______________________________________________ Numpy-discussion mailing list Numpy-discuss...@scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Sep 30, 2008 at 12:53, Brian Blais <bblais@bryant.edu> wrote:
Hello, I am trying to use xml-rpc to be able to run some simulations remotely. I am running into a problem with the transfer of numpy arrays. my server code looks like: #!/usr/bin/env python def again(x): # test out the sending of data return [x,x]
from SimpleXMLRPCServer import SimpleXMLRPCServer SimpleXMLRPCServer.allow_reuse_address = 1 server = SimpleXMLRPCServer(("", 8000)) server.register_function(again) try: print "Serving..." server.serve_forever() # Start the server finally: print "done." server.server_close()
my client code looks like: import numpy from xmlrpclib import ServerProxy server=ServerProxy('http://localhost:8000') server.again(5) # this works b=numpy.random.rand(5,5) server.again(b) # this gives an error this gives the error: <type 'exceptions.TypeError'>: cannot marshal <type 'numpy.ndarray'> objects which seems to be a deficiency of the marshal library, or perhaps I am doing something wrong. Is there a way to fix this? Is there another approach that I should be using?
The marshal module *only* handles builtin Python types. It explicitly does not handle anything from third parties like numpy. As Uwe suggests, you can use tostring()/fromstring() yourself and also pass along the dtype and shape information. With numpy 1.2.0, there is a somewhat better alternative. I have defined a small file format that should represent all numpy arrays with dtype and shape information embedded in its header. That way, you just have to pass the string. from cStringIO import StringIO from numpy.lib import format def from_string(s): f = StringIO(s) arr = format.read_array(f) return arr def to_string(arr): f = StringIO() format.write_array(f, arr) s = f.getvalue() return s -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Hi, [..snip..] On Tuesday 30 September 2008 12:05:27 Robert Kern wrote:
doing something wrong. Is there a way to fix this? Is there another approach that I should be using?
The marshal module *only* handles builtin Python types. It explicitly does not handle anything from third parties like numpy.
but you can use the copy_reg module ('copyreg' in py2.6) to register pickling/unpickling functions to make 'pickle' able to handle user-types: ## test_pickling_numpy.py ## import numpy import copy_reg import cPickle as pickle def from_string(s): return numpy.fromstring(s) def to_string(o): return from_string, (o.tostring(),) a = numpy.arange (10,dtype=float) copy_reg.pickle (numpy.ndarray, to_string, from_string) s = pickle.dumps (a) b = pickle.loads (s) assert (a==b, "pickling failed !") ## EOF ## see for more informations: http://effbot.org/librarybook/copy-reg.htm cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ###################################
Sebastien, numpy arrays are picklable; so no need to register them with copy_reg. I believe the actual problem with xmlrpclib is that it uses the marshal protocol (only supports core builtin types), and not the pickle protocol. On Tue, Sep 30, 2008 at 5:18 PM, Sebastien Binet <hep.sebastien.binet@gmail.com> wrote:
Hi,
[..snip..] On Tuesday 30 September 2008 12:05:27 Robert Kern wrote:
doing something wrong. Is there a way to fix this? Is there another approach that I should be using?
The marshal module *only* handles builtin Python types. It explicitly does not handle anything from third parties like numpy.
but you can use the copy_reg module ('copyreg' in py2.6) to register pickling/unpickling functions to make 'pickle' able to handle user-types:
## test_pickling_numpy.py ## import numpy import copy_reg import cPickle as pickle
def from_string(s): return numpy.fromstring(s)
def to_string(o): return from_string, (o.tostring(),)
a = numpy.arange (10,dtype=float) copy_reg.pickle (numpy.ndarray, to_string, from_string) s = pickle.dumps (a) b = pickle.loads (s) assert (a==b, "pickling failed !") ## EOF ##
see for more informations: http://effbot.org/librarybook/copy-reg.htm
cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ###################################
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
-- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Lisandro, On Tuesday 30 September 2008 15:24:56 Lisandro Dalcin wrote:
Sebastien, numpy arrays are picklable; so no need to register them with copy_reg. I believe the actual problem with xmlrpclib is that it uses the marshal protocol (only supports core builtin types), and not the pickle protocol. yeah... Robert pointed it to me that as xmlrpc is meant for cross-language RPC, sending python objects over the wire isn't so useful, hence the usage of marshal instead of pickle.
cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ###################################
On Sep 30, 2008, at 18:42 , Sebastien Binet wrote:
yeah... Robert pointed it to me that as xmlrpc is meant for cross- language RPC, sending python objects over the wire isn't so useful, hence the usage of marshal instead of pickle.
thanks for all of the help. My initial solution is to pickle my object, with the text-based version of pickle, and send it across rpc. I do this because the actual thing I am sending is a dictionary, with lots of arrays, and other things. I'll have a look at the format that Robert sent, because that looks useful for other things I am doing. Sebastien, why is sending python objects over the wire not so useful? is there a better way to do this sort of thing than xmlrpc? I thought it looked particularly simple (other than this pickling issue, of course. :) ). thanks, bb -- Brian Blais bblais@bryant.edu http://web.bryant.edu/~bblais
Brian, On Tuesday 30 September 2008 17:27:32 Brian Blais wrote:
On Sep 30, 2008, at 18:42 , Sebastien Binet wrote:
yeah... Robert pointed it to me that as xmlrpc is meant for cross- language RPC, sending python objects over the wire isn't so useful, hence the usage of marshal instead of pickle.
thanks for all of the help. My initial solution is to pickle my object, with the text-based version of pickle, and send it across rpc. I do this because the actual thing I am sending is a dictionary, with lots of arrays, and other things. I'll have a look at the format that Robert sent, because that looks useful for other things I am doing.
Sebastien, why is sending python objects over the wire not so useful? it isn't useful if you don't have a python interpreter instance on the other end of the wire, ie a python program on one end and a javascript one on the other end :) (well, you can always re-engineer the pickle format to unpickle what you received, etc... it is 'just' a matter of programming)
cheers, sebastien. -- ################################### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ###################################
On Tue, Sep 30, 2008 at 9:27 PM, Brian Blais <bblais@bryant.edu>
thanks for all of the help. My initial solution is to pickle my object, with the text-based version of pickle, and send it across rpc. I do this because the actual thing I am sending is a dictionary, with lots of arrays, and other things. I'll have a look at the format that Robert sent, because that looks useful for other things I am doing.
Did you try to send binary pickles (protocol=2)? Perhaps it works, give a try! Of course, you need the client and server machines having the same arch.
Sebastien, why is sending python objects over the wire not so useful? is there a better way to do this sort of thing than xmlrpc? I thought it looked particularly simple (other than this pickling issue, of course. :) ).
I believe xmlrpclib is currently the simpler approach. Some day I'll have the time to implement something similar using MPI communication with mpi4py. However, I believe it can be done even better: local, client-side proxies should automatically provide access to all members/methods of remote, server-side instances. The registering step needed within xmlrpclib is a bit ugly ;-) -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Lisandro Dalcin wrote:
I believe xmlrpclib is currently the simpler approach. Some day I'll have the time to implement something similar using MPI communication with mpi4py. However, I believe it can be done even better: local, client-side proxies should automatically provide access to all members/methods of remote, server-side instances. The registering step needed within xmlrpclib is a bit ugly ;-)
Try pyro: <http://pyro.sourceforge.net/> or rpyc: <http://rpyc.wikispaces.com/>, both of which if I recall correctly, do implement this. VS.
On Sep 30, 2008, at 23:16 , Lisandro Dalcin wrote:
On Tue, Sep 30, 2008 at 9:27 PM, Brian Blais <bblais@bryant.edu>
thanks for all of the help. My initial solution is to pickle my object, with the text-based version of pickle, and send it across rpc. I do this because the actual thing I am sending is a dictionary, with lots of arrays, and other things. I'll have a look at the format that Robert sent, because that looks useful for other things I am doing.
Did you try to send binary pickles (protocol=2)? Perhaps it works, give a try! Of course, you need the client and server machines having the same arch.
I tried that first, and marshal doesn't handle binary streams, so that's when I tried the text version. I'm not sending a lot of info, so it's not that big of a deal. Most of the time is spent on the simulation on the other end. bb -- Brian Blais bblais@bryant.edu http://web.bryant.edu/~bblais
participants (6)
-
Brian Blais -
Lisandro Dalcin -
Robert Kern -
Sebastien Binet -
Uwe Schmitt -
Vincent Schut