[IPython-dev] DAG Dependencies

Satrajit Ghosh satra at mit.edu
Thu Oct 28 10:50:10 EDT 2010


hi min,

this is great. a few things that might be useful to consider:

* optionally offload the dag directly to the underlying scheduler if it has
dependency support (i.e., SGE, Torque/PBS, LSF)
* something we currently do in nipype is that we provide a configurable
option to continue processing if a given node fails. we simply remove the
dependencies of the node from further execution and generate a report at the
end saying which nodes crashed.
* callback support for node: node_started_cb, node_finished_cb
* support for nodes themselves being DAGs
* the concept of stash and pop for DAG nodes. i.e. a node which is a dag can
stash itself while it's internal nodes execute and should not take up any
engine.

also i was recently with some folks who have been using DRMAA (
http://en.wikipedia.org/wiki/DRMAA) as the underlying common layer for
communicating with PBS, SGE, LSF, Condor. it might be worthwhile taking a
look (if you already haven't) to see what sort of mechanisms might help you.
a python binding is available at:
http://code.google.com/p/drmaa-python/wiki/Tutorial

cheers,

satra


On Thu, Oct 28, 2010 at 3:57 AM, MinRK <benjaminrk at gmail.com> wrote:

> Hello,
>
> In order to test/demonstrate arbitrary DAG dependency support in the new
> ZMQ Python scheduler, I wrote an example using NetworkX, as Fernando
> suggested.
>
> It generates a random DAG with a given number of nodes and edges, runs a
> set of empty jobs (one for each node) using the DAG as a dependency graph,
> where each edge represents a job depending on another.
> It then validates the results, ensuring that no job ran before its
> dependencies, and draws the graph, with nodes arranged in X according to
> time, which means that all arrows must point to the right if the
> time-dependencies were met.
>
> It happily handles pretty elaborate (hundreds of edges) graphs.
>
> Too bad I didn't have this done for today's Py4Science talk.
>
>  Script can be found here:
> http://github.com/minrk/ipython/blob/newparallel/examples/demo/dagdeps.py
>
> -MinRK
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101028/ce6cab6f/attachment.html>


More information about the IPython-dev mailing list