Aureli Soria Frisch wrote:
Hi all,
I am trying to make run a numerical computation (with arrays) in different computers simultaneously (in parallel). The computation is done under Linux.
For that purpose a master organizes the process and send rexec (remote execute) commands to the different slaves via the python command spawnlp. The slaves execute the script specified through rexec.
Inside this script the slaves open a file with the arguments of the process, which were serialized via pickle, then make the numerical computation, and write the result (a NumPy array) again via pickle in a file. This file is opened by the master, which uses the different results.
I am having the problem that the master sometimes (the problem does not happen always!!!) open the result and load an object of
instead of the expected object of (what then produces an error). I have tested the type of the objects in the slaves and it is always 'array'. Has someone made similar experiences by 'pickling' arrays? Could it be a problem of the different computers running versions of Python from 2.0 to 2.2.1? Or a problem of different versions of NumPy?
Is there any other way for doing such a parallel computation?
I am not sure what is causing the unpickling problem you are seeing, but I suggest that you consider MPI for what you are doing. There are a number of Python MPI interfaces around, but I can personally recommend PyPar by Ole Nielsen at the Australian National University. You can use PyPar with LAM/MPI, which runs in user mode and is very easy to install, and PyPar itself does not require any modifications to the Python interpreter. PyPar will automatically serialise Python objects for you (and deserialise them at the destination) but also has methods to send NumPy arrays directly which is very efficient. See http://datamining.anu.edu.au/~ole/pypar/ for more details. Tim C