[IPython-dev] newparallel

MinRK benjaminrk at gmail.com
Sun Feb 6 18:42:40 EST 2011


On Sun, Feb 6, 2011 at 12:47, Satrajit Ghosh <satra at mit.edu> wrote:
> hi min, brian and fernando,
>
> i just checked out the newparallel branch from the github repo. a few
> questions:
>
> 1. is this the best place to look at documentation?
>
> http://minrk.github.com/ipython/parallelz/index.html

Yes, but I only just started updating those to match the newparallel
code, so they aren't fully up to date.

>
> (unfortunately, i could not compile the docs on my 64-bit osx machine
> without wx.)

wxpython 2.9 now supports cocoa, but if you want to use it with the
System default Python, you need to build from source (which takes
ages).

>
> 2. is dag support already available in newparallel? i.e., can i add tasks
> with dependencies before executing them? if not, where is the best place for
> hacking this in?

Yes, you can implement a DAG of dependencies, as seen here:
https://github.com/ipython/ipython/blob/newparallel/examples/demo/dag/dagdeps.py.

Note that you must use one of the Python schedulers (via '--scheme
lru' to ipcontrollerz), because the default pure-zmq scheduler ignores
dependencies.

There is probably some more work to go into Dependencies, specifically
regarding how to interpret job failures and unreachable DAG nodes.

>
> 3. fernando: is your workflow branch mergeable with this? the ability to
> spawn and shutdown engines as needed (especially in the context of clusters
> such as SGE) has come up for discussion on the nipype side a fair bit in the
> last few weeks.

The last commit of that branch has not been merged yet, but over the
last week I have updated the newparallel scripts to use the IPython
config system, so starting an engine is no longer the same as
represented in those scripts, but it should be easy to update them.

>
> 4. is there a good place to read about the parallel architecture?

The parallel messaging/connections documents are the best description
we have of how the underlying architecture works, but they still need
to be fleshed out:

http://minrk.github.com/ipython/development/parallel_messages.html

-MinRK

>
> cheers,
>
> satra
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>



More information about the IPython-dev mailing list