![](https://secure.gravatar.com/avatar/86776c6c595af5117de5ba7b41bc33b5.jpg?s=120&d=mm&r=g)
One quick question: Why does the MA module have an average function, but not Numeric? And what is the equivalent in numarray? -- Magnus Lie Hetland The Anygui Project http://hetland.org http://anygui.org
![](https://secure.gravatar.com/avatar/96bc2a659fff9486fa58f17fd4f95952.jpg?s=120&d=mm&r=g)
Hi all, I am trying to make run a numerical computation (with arrays) in different computers simultaneously (in parallel). The computation is done under Linux. For that purpose a master organizes the process and send rexec (remote execute) commands to the different slaves via the python command spawnlp. The slaves execute the script specified through rexec. Inside this script the slaves open a file with the arguments of the process, which were serialized via pickle, then make the numerical computation, and write the result (a NumPy array) again via pickle in a file. This file is opened by the master, which uses the different results. I am having the problem that the master sometimes (the problem does not happen always!!!) open the result and load an object of <type 'instance'> instead of the expected object of <type 'array'> (what then produces an error). I have tested the type of the objects in the slaves and it is always 'array'. Has someone made similar experiences by 'pickling' arrays? Could it be a problem of the different computers running versions of Python from 2.0 to 2.2.1? Or a problem of different versions of NumPy? Is there any other way for doing such a parallel computation? Thanks for the time... Regards, Aureli -- ################################# Aureli Soria Frisch Fraunhofer IPK Dept. Pattern Recognition post: Pascalstr. 8-9, 10587 Berlin, Germany e-mail: aureli@ipk.fhg.de fon: +49 30 39006-143 fax: +49 30 3917517 web: http://vision.fhg.de/~aureli/web-aureli_en.html #################################
![](https://secure.gravatar.com/avatar/9567eaa720c05c4f4a2dbf7d831f1ef8.jpg?s=120&d=mm&r=g)
Aureli Soria Frisch wrote:
I am not sure what is causing the unpickling problem you are seeing, but I suggest that you consider MPI for what you are doing. There are a number of Python MPI interfaces around, but I can personally recommend PyPar by Ole Nielsen at the Australian National University. You can use PyPar with LAM/MPI, which runs in user mode and is very easy to install, and PyPar itself does not require any modifications to the Python interpreter. PyPar will automatically serialise Python objects for you (and deserialise them at the destination) but also has methods to send NumPy arrays directly which is very efficient. See http://datamining.anu.edu.au/~ole/pypar/ for more details. Tim C
![](https://secure.gravatar.com/avatar/eb281ac8437ba6df4ef5f0f9686e7c3e.jpg?s=120&d=mm&r=g)
Aureli Soria Frisch <Aureli.Soria_Frisch@ipk.fhg.de> writes:
Yes -- pickling isn't meant to work across different python versions (it might to some extent, but I wouldn't try it unless there is no way around it). Using netcdf as a data format instead of pickling might also be a solution (if intermediate storage on the disk is not too inefficient, but your original approach involved that anyway). Konrad Hinsen has written a nice wrapper for python that is quite easy to use: http://starship.python.net/crew/hinsen/scientific.html. alex -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.Schmolck@gmx.net http://www.dcs.ex.ac.uk/people/aschmolc/
![](https://secure.gravatar.com/avatar/36b41384e4f45dcf821fdeb980593534.jpg?s=120&d=mm&r=g)
Hi all, Has someone implemented a function for arrays that behaves like the index(*) method for lists (it should then consider something like a tolerance parameter). I suppose it could be maybe done with array.tolist() and list.index(), but have someone implemented something more elegant/array-based? Thanks in advance Aureli PD: (*) index receive a value as an argument and retunrs the index of the list member equal to this value... -- ################################# Aureli Soria Frisch Fraunhofer IPK Dept. Pattern Recognition post: Pascalstr. 8-9, 10587 Berlin, Germany e-mail: aureli@ipk.fhg.de fon: +49 30 39006-143 fax: +49 30 3917517 web: http://vision.fhg.de/~aureli/web-aureli_en.html #################################
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
Aureli Soria Frisch wrote:
I think the basics of what you're looking for are something like: def index(a, b, eps): return nonzero(abs(a-b) < eps) which should return all indices at which the absolute value of the difference between elements of a and b differ by less than eps. e.g.:
Todd -- Todd Miller jmiller@stsci.edu STSCI / SSG (410) 338 4576
![](https://secure.gravatar.com/avatar/96bc2a659fff9486fa58f17fd4f95952.jpg?s=120&d=mm&r=g)
Hi all, I am trying to make run a numerical computation (with arrays) in different computers simultaneously (in parallel). The computation is done under Linux. For that purpose a master organizes the process and send rexec (remote execute) commands to the different slaves via the python command spawnlp. The slaves execute the script specified through rexec. Inside this script the slaves open a file with the arguments of the process, which were serialized via pickle, then make the numerical computation, and write the result (a NumPy array) again via pickle in a file. This file is opened by the master, which uses the different results. I am having the problem that the master sometimes (the problem does not happen always!!!) open the result and load an object of <type 'instance'> instead of the expected object of <type 'array'> (what then produces an error). I have tested the type of the objects in the slaves and it is always 'array'. Has someone made similar experiences by 'pickling' arrays? Could it be a problem of the different computers running versions of Python from 2.0 to 2.2.1? Or a problem of different versions of NumPy? Is there any other way for doing such a parallel computation? Thanks for the time... Regards, Aureli -- ################################# Aureli Soria Frisch Fraunhofer IPK Dept. Pattern Recognition post: Pascalstr. 8-9, 10587 Berlin, Germany e-mail: aureli@ipk.fhg.de fon: +49 30 39006-143 fax: +49 30 3917517 web: http://vision.fhg.de/~aureli/web-aureli_en.html #################################
![](https://secure.gravatar.com/avatar/9567eaa720c05c4f4a2dbf7d831f1ef8.jpg?s=120&d=mm&r=g)
Aureli Soria Frisch wrote:
I am not sure what is causing the unpickling problem you are seeing, but I suggest that you consider MPI for what you are doing. There are a number of Python MPI interfaces around, but I can personally recommend PyPar by Ole Nielsen at the Australian National University. You can use PyPar with LAM/MPI, which runs in user mode and is very easy to install, and PyPar itself does not require any modifications to the Python interpreter. PyPar will automatically serialise Python objects for you (and deserialise them at the destination) but also has methods to send NumPy arrays directly which is very efficient. See http://datamining.anu.edu.au/~ole/pypar/ for more details. Tim C
![](https://secure.gravatar.com/avatar/eb281ac8437ba6df4ef5f0f9686e7c3e.jpg?s=120&d=mm&r=g)
Aureli Soria Frisch <Aureli.Soria_Frisch@ipk.fhg.de> writes:
Yes -- pickling isn't meant to work across different python versions (it might to some extent, but I wouldn't try it unless there is no way around it). Using netcdf as a data format instead of pickling might also be a solution (if intermediate storage on the disk is not too inefficient, but your original approach involved that anyway). Konrad Hinsen has written a nice wrapper for python that is quite easy to use: http://starship.python.net/crew/hinsen/scientific.html. alex -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.Schmolck@gmx.net http://www.dcs.ex.ac.uk/people/aschmolc/
![](https://secure.gravatar.com/avatar/36b41384e4f45dcf821fdeb980593534.jpg?s=120&d=mm&r=g)
Hi all, Has someone implemented a function for arrays that behaves like the index(*) method for lists (it should then consider something like a tolerance parameter). I suppose it could be maybe done with array.tolist() and list.index(), but have someone implemented something more elegant/array-based? Thanks in advance Aureli PD: (*) index receive a value as an argument and retunrs the index of the list member equal to this value... -- ################################# Aureli Soria Frisch Fraunhofer IPK Dept. Pattern Recognition post: Pascalstr. 8-9, 10587 Berlin, Germany e-mail: aureli@ipk.fhg.de fon: +49 30 39006-143 fax: +49 30 3917517 web: http://vision.fhg.de/~aureli/web-aureli_en.html #################################
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
Aureli Soria Frisch wrote:
I think the basics of what you're looking for are something like: def index(a, b, eps): return nonzero(abs(a-b) < eps) which should return all indices at which the absolute value of the difference between elements of a and b differ by less than eps. e.g.:
Todd -- Todd Miller jmiller@stsci.edu STSCI / SSG (410) 338 4576
participants (6)
-
Alexander Schmolck
-
Aureli Soria Frisch
-
Aureli Soria Frisch
-
Magnus Lie Hetland
-
Tim Churches
-
Todd Miller