[IPython-dev] DAG Dependencies

[Brian Granger]

>> here is the scenario where this becomes a useful thing (and hence to
>> optionally have it). let's say under sge usage you have started 10
>> clients/ipengines. now at the time of creating the clients one machine with
>> 10 allocations was free and sge routed all the 10 clients to that machine.
>> now this will be the machine that will be used for all ipcluster processing.
>> whereas if the node distribution and ipengine startup were to happen
>> simultaneously at the level of the sge scheduler, processes would get routed
>> to the best available slot at the time of execution.
>
> You should get this for free if our ipcluster script on SGE/etc. can
> grow/shrink to fit available resources.  Remember, jobs don't get submitted
> to engines until their dependencies are met.  It handles new engines coming
> just fine (still some work to do to handle engines disappearing gracefully).
> Engines should correspond to actual available resources, and we should
> probably have an ipcluster script that supports changing resources on a
> grid, but I'm not so sure about the scheduler itself.

And I do think that having ipcluster be able to grow/shrink the
cluster is an important feature.

>> perhaps i misunderstood what happens in the current implementation. if you
>> have a DAG such as (A,B) (B,E) (A,C) (C,D) and let's say C fails, does the
>> current dag controller continue executing B,E? or does it crash at the first
>> failure. we have the option to go either way in nipype. if something
>> crashes, stop or if something crashes, process all things that are not
>> dependent on the crash.
>
> The Twisted Scheduler considers a task dependency unmet if the task raised
> an error, and currently the ZMQ scheduler has no sense of error/success, so
> it works the other way, but can easily be changed if I add the ok/error
> status to the msg header (I should probably do this).  I think this is a
> fairly reasonable use case to have a switch on failure/success, and it would
> actually get you the callbacks you mention:
> msg_id = client.apply(job1)
> client.apply(cleanup, follow_failure=msg_id)
> client.apply(job2, after_success=msg_id)
> client.apply(job3, after_all=msg_id)
> With this:
>     iff job1 fails: cleanup will be run *in the same place* as job1
>     iff job1 succeeds: job2 will be run somewhere
>     when job1 finishes: job3 will be run somewhere, regardless of job1's
> status
> Satrajit: Does that sound adequate?
> Brian: Does that sound too complex?

I see a couple of different options if a task in a DAG fails:

* It is fatal.  The task failed and is absolutely required by
subsequent tasks, and nothing can be done.  In this case, I think the
tasks that depend on the failed one should be aborted.

* It is not fatal.  In this case, something can be done to remedy the
situation.  Maybe a tasks couldn't find the data it needs, but it
could find it elsewhere.  Or maybe the failed task has to close some
resource.  But, in all of these situations, I would say that all of
this logic should be built into the task itself (using regular Python
exception handling).  IOW, tasks should handle non-fatal errors
themselves.

But....this analysis doesn't make sense if there are task failure
modes that can't be handled by the tasks itself by catching
exceptions, etc.

My main concern with the above approach is that it adds complexity
both to the public APIs, but more importantly to the scheduler itself.






> A DAG-node is really the same has having the root(s) of a sub-DAG have the
> dependencies of the DAG-node, and anything that would depend on the DAG-node
> actually depends on any|all of the termini of the sub-DAG, no?

That is my thinking.

Brian

-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com