[IPython-dev] Parallel: Report progress from the engines

Fernando Perez fperez.net at gmail.com
Mon Aug 12 19:34:29 EDT 2013


Hi,

these two notebooks provide examples of monitoring a parallel run (in
this case an MPI one, but you can adapt this to your use case). The
first uses engine-driven data publication, the second uses client-side
polling in a thread; each approach has its pros and cons:

http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/examples/parallel/InteractiveMPI-publish-data.ipynb

http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/examples/parallel/InteractiveMPI.ipynb

And here is a bit more info on the structure of async results that can
be used for timing/monitoring:

http://ipython.org/ipython-doc/rel-1.0.0/parallel/asyncresult.html#timing

I seem to recall Min had some more examples of this, but can't seem to
find them right now.

Cheers,

f

On Mon, Aug 12, 2013 at 1:50 PM, Darlan Cavalcante Moreira
<darcamo at gmail.com> wrote:
>
> Hi list,
>
> I'm using the fantastic parallel computing machinery from IPython, which
> works really well. However, the simulations I'm performing take a few
> hours to finish and I'd like to see the progress from each engine. The
> number of tasks is small (usually from five to eight) such that only
> knowing how many tasks have finished is not very useful for me. I need
> to track the progress of each individual task.
>
> To give an example, with the multiprocessing module I could create a
> Queue that I can pass as argument to the task that will run in the other
> process. My tasks are basically a for loop running a few lines of code
> in each iteration. Therefore, the task can put the iteration number in
> the Queue indicating its progress, which is then read (every 5 seconds)
> from another process that finally prints the progress in the terminal.
>
> However, I have no idea how I can do this with the IPython parallel
> machinery. Is there any way for the engines to send data back (to the
> controller?) that I can use to track their progress?
>
>
> --
> Darlan Cavalcante
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



-- 
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail



More information about the IPython-dev mailing list