[IPython-dev] Parallel: Report progress from the engines

Darlan Cavalcante Moreira darcamo at gmail.com
Tue Aug 13 23:07:20 EDT 2013

benjaminrk at gmail.com writes:

Thank you Fernando and MinRK.

I haven't used MPI before. I'll need some time to dig through these
examples, but it's a good start.

I tried to run the notebooks, but I had a problem with the line below

    from IPython.kernel.zmq.datapub import publish_data

which gives me an import error, since IPython.kernel is not used
anymore. I could not find out where this function was moved to. Googling
for publish_data I found the notebook below
which confirms this function is exactly what I was looking for (as long
as I can get the value of the published variable while the task is still
running). Is it still in IPython?

For now MinRK suggestion seems like a good workaround. I'll have to
change my code to use asynchronous tasks instead of sync tasks as I do
now, but that is easy enough.


> You can also monitor the progress with simple print statements by viewing
> ar.stdout <https://gist.github.com/minrk/6171348>
> On Mon, Aug 12, 2013 at 4:34 PM, Fernando Perez <fperez.net at gmail.com>wrote:
>> Hi,
>> these two notebooks provide examples of monitoring a parallel run (in
>> this case an MPI one, but you can adapt this to your use case). The
>> first uses engine-driven data publication, the second uses client-side
>> polling in a thread; each approach has its pros and cons:
>> http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/examples/parallel/InteractiveMPI-publish-data.ipynb
>> http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/examples/parallel/InteractiveMPI.ipynb
>> And here is a bit more info on the structure of async results that can
>> be used for timing/monitoring:
>> http://ipython.org/ipython-doc/rel-1.0.0/parallel/asyncresult.html#timing
>> I seem to recall Min had some more examples of this, but can't seem to
>> find them right now.
>> Cheers,
>> f
>> On Mon, Aug 12, 2013 at 1:50 PM, Darlan Cavalcante Moreira
>> <darcamo at gmail.com> wrote:
>> >
>> > Hi list,
>> >
>> > I'm using the fantastic parallel computing machinery from IPython, which
>> > works really well. However, the simulations I'm performing take a few
>> > hours to finish and I'd like to see the progress from each engine. The
>> > number of tasks is small (usually from five to eight) such that only
>> > knowing how many tasks have finished is not very useful for me. I need
>> > to track the progress of each individual task.
>> >
>> > To give an example, with the multiprocessing module I could create a
>> > Queue that I can pass as argument to the task that will run in the other
>> > process. My tasks are basically a for loop running a few lines of code
>> > in each iteration. Therefore, the task can put the iteration number in
>> > the Queue indicating its progress, which is then read (every 5 seconds)
>> > from another process that finally prints the progress in the terminal.
>> >
>> > However, I have no idea how I can do this with the IPython parallel
>> > machinery. Is there any way for the engines to send data back (to the
>> > controller?) that I can use to track their progress?
>> >
>> >
>> > --
>> > Darlan Cavalcante
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> --
>> Fernando Perez (@fperez_org; http://fperez.org)
>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
>> fernando.perez-at-berkeley: contact me here for any direct mail
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

Darlan Cavalcante Moreira
darcamo at gmail.com

More information about the IPython-dev mailing list