[IPython-dev] ipython + MPI

[Marek Wojciechowski]

Hi!
A while ago I have written a small patch to the last ipython (7.2) which  
consist mainly in writing the following function:

def mpi_raw_input(prompt):
     rank = 0; mpi = False; size = 1
     try:
         from mpi4py import MPI
         mpi = True
         COMM = MPI.COMM_WORLD
         rank = MPI.rank
	  size = MPI.size
	  line = ''
     except: pass
     sys.stdout.flush() #Just to avoid messy output
     if rank == 0:
         line = raw_input(prompt)  # Command is read by first process only
     if mpi and size > 1: line = COMM.Bcast(line, 0)  # And is broadcasted  
to all processes involved
     return line

Then at the top of iplib.py I made an assignment:  raw_input_original =  
mpi_raw_input
instead of:  raw_input_original = raw_input

I'm using mpi4py package here from http://www.cimec.org.ar/python, but any  
mpi port to python could be used here.
This slight change allows to run ipython with simple:
     mpirun -np 4 ipython (assuming you have modern enough MPI  
implementation)
All commands are read by first process only and broadcasted to all others  
for execution. With this it is possible to follow plain MPI approach to  
parallel execution:

[In]<1> 1+3
[Out][Proc0]<1> 4
[Out][Proc1]<1> 4
[Out][Proc3]<1> 4
[Out][Proc2]<1> 4

[In]<2> from mpi4py import MPI
[In]<3> if MPI.rank in [1, 3]: print "I'm process %i" %(MPI.rank)
I'm process 3
I'm process 1

[In]<5> from math import sin
[In]<6> bg sin(3)  # ipython magic works as well :)
Starting job # 0 in a separate thread.
Starting job # 0 in a separate thread.
Starting job # 0 in a separate thread.
Starting job # 0 in a separate thread.

Obviously, there are problems with standard output of ipython processes,  
as we are not always interested in output of all of them. However this can  
be handled in a similar way as the input, and was partially done (i made  
for example help messages produced by ipython to be displayed at first  
process only).

Now i observed that new ipython development introduces a parallel  
execution via "controllers" and "engines". Is the above very simple  
approach comparable in any way with this new development. How does it  
support MPI communication? What are the other advanteges of the new kernel  
above the current ipython?

Thanks for any replay

-- 
Marek Wojciechowski