[Numpy-discussion] Best way to run python parallel

[Lisandro Dalcin]

On 3/29/07, Brad Malone <brad.malone at gmail.com> wrote:
> Hi, I use python for some fairly heavy scientific computations (at least to
> be running on a single processor) and would like to use it in parallel.
> I've seen some stuff online about Parallel Python and mpiPy, but I don't
> know much about them.  Is a python-specific program needed to run python in
> parallel or are the others ( e.g., mpi/poe) just more difficult to work
> with?  And which one would you recommend?

Brad, I am the author of mpi4py. I really suggest you try this for
several reasons:

- As Brian Granger said, it build almost anywhere (I only use linux,
but other guys use it on MAC and Win).

- It is just a python module, you don't need to rebuild a python
interpreter (really true unless you want to use old MPICH1
implementation). However, parallel debuggin can be really hard. I am
waiting for Brian's work on IPython1.

- You can communicate numpy arrays very efficiently (for moderately
sized arrays, the performace is almost the same that the one in C on
gigabit ethernet clusters). In the current state, you have to manually
specify the MPI datatypes. This could be automated in the future, once
the Travis's buffer interface gets in on core Python. Additionally,
you can communicate almost any object supporting pickling, this is not
so efficient, but really transparent is you want to communicate a
small, nested object like a dict.

- And finally, one thing that is really important to me (and in fact,
what motivated my to develop mpi4py). Its API is really similar to the
C++ MPI bindings, so you can read any MPI book/tutorial/code snipet,
and translate it to working Python code.

A final note. Just in case you need to do parallel linear algebra, and
solve linear/nonlinear systems of ecuations, I've also developed
petsc4py, a port to PETSc libraries.

You can find mpi4py (and also petsc4py) at PyPI for download (or just
try setuptools 'easy_install', this will download, build and install
mpi4py for you, just be sure of having 'mpicc' MPI compiler wrappers
on your PATH).

-- 
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594