[IPython-dev] Parallel: Report progress from the engines

[MinRK]

You can also monitor the progress with simple print statements by viewing
ar.stdout <https://gist.github.com/minrk/6171348>


On Mon, Aug 12, 2013 at 4:34 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> Hi,
>
> these two notebooks provide examples of monitoring a parallel run (in
> this case an MPI one, but you can adapt this to your use case). The
> first uses engine-driven data publication, the second uses client-side
> polling in a thread; each approach has its pros and cons:
>
>
> http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/examples/parallel/InteractiveMPI-publish-data.ipynb
>
>
> http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/examples/parallel/InteractiveMPI.ipynb
>
> And here is a bit more info on the structure of async results that can
> be used for timing/monitoring:
>
> http://ipython.org/ipython-doc/rel-1.0.0/parallel/asyncresult.html#timing
>
> I seem to recall Min had some more examples of this, but can't seem to
> find them right now.
>
> Cheers,
>
> f
>
> On Mon, Aug 12, 2013 at 1:50 PM, Darlan Cavalcante Moreira
> <darcamo at gmail.com> wrote:
> >
> > Hi list,
> >
> > I'm using the fantastic parallel computing machinery from IPython, which
> > works really well. However, the simulations I'm performing take a few
> > hours to finish and I'd like to see the progress from each engine. The
> > number of tasks is small (usually from five to eight) such that only
> > knowing how many tasks have finished is not very useful for me. I need
> > to track the progress of each individual task.
> >
> > To give an example, with the multiprocessing module I could create a
> > Queue that I can pass as argument to the task that will run in the other
> > process. My tasks are basically a for loop running a few lines of code
> > in each iteration. Therefore, the task can put the iteration number in
> > the Queue indicating its progress, which is then read (every 5 seconds)
> > from another process that finally prints the progress in the terminal.
> >
> > However, I have no idea how I can do this with the IPython parallel
> > machinery. Is there any way for the engines to send data back (to the
> > controller?) that I can use to track their progress?
> >
> >
> > --
> > Darlan Cavalcante
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> --
> Fernando Perez (@fperez_org; http://fperez.org)
> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> fernando.perez-at-berkeley: contact me here for any direct mail
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130813/0dbc000d/attachment.html>