[IPython-dev] Parallel: Report progress from the engines

Fernando Perez fperez.net at gmail.com
Mon Aug 12 19:34:29 EDT 2013


these two notebooks provide examples of monitoring a parallel run (in
this case an MPI one, but you can adapt this to your use case). The
first uses engine-driven data publication, the second uses client-side
polling in a thread; each approach has its pros and cons:



And here is a bit more info on the structure of async results that can
be used for timing/monitoring:


I seem to recall Min had some more examples of this, but can't seem to
find them right now.



On Mon, Aug 12, 2013 at 1:50 PM, Darlan Cavalcante Moreira
<darcamo at gmail.com> wrote:
> Hi list,
> I'm using the fantastic parallel computing machinery from IPython, which
> works really well. However, the simulations I'm performing take a few
> hours to finish and I'd like to see the progress from each engine. The
> number of tasks is small (usually from five to eight) such that only
> knowing how many tasks have finished is not very useful for me. I need
> to track the progress of each individual task.
> To give an example, with the multiprocessing module I could create a
> Queue that I can pass as argument to the task that will run in the other
> process. My tasks are basically a for loop running a few lines of code
> in each iteration. Therefore, the task can put the iteration number in
> the Queue indicating its progress, which is then read (every 5 seconds)
> from another process that finally prints the progress in the terminal.
> However, I have no idea how I can do this with the IPython parallel
> machinery. Is there any way for the engines to send data back (to the
> controller?) that I can use to track their progress?
> --
> Darlan Cavalcante
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

More information about the IPython-dev mailing list