[IPython-dev] taskclient clear and hierarchical parallel processing

[Satrajit Ghosh]

hi brian,

For now it is.  The issue is that the controller stores all the tasks
> in memory (eventually it should store them on disk), so clear() is
> used to get rid of tasks you are done with so you can save memory.  So
> yes, for now it is the desired behavior.
>

ah. i haven't actually performed a memory consumption analysis, but clearing
things when possible seemed like a reasonable thing to do. again the key
problem was running two or more parallel analysis scripts simultaneously. we
have situations where the workflow execution graph contains close to 700
nodes.


> > alternatively, is there a way to clear a specific task based on its task
> id?
> > basically i want to use the same pool of resources from multiple client
> > connections.
>
> We don't have a way of doing this currently.  Basically, you should
> call clear() when you know all clients are done with a set of tasks
> and you want to free up that memory.
>

any suggestions on how to figure out when all clients are done?



> >
> > now i can flatten out all the DAGs and run them, but it would be neat if
> > there was a pattern that would enable running these as concrete entities.
>
> I have not really thought about this point before.  But I guess that
> you can get into a deadlock situation if all the nodes are busy and
> the sub-DAGs can't be executed because their parents are using all the
> engines.  Obviously if you have a larger number of engines, this
> problem can be avoided, but it would be nice if the scheduler itself
> could handle this.  Some questions:
>
> * Could be re-write it so that the sub DAGs were top-level tasks
> rather than doing the recursive task-submit-a-task thing?
>

i think this is what i meant by flattening out the DAGs and that's what we
are currently doing.


> We might have to look into ways of allowing tasks to be paused to that
> sub-tasks can run while the parent is paused.  Interesting things to
> think about...
>

that would be neat. it's like the the node saying, put me to sleep and use
my resources, when my child tasks are done wake me up.

cheers,

satra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100513/55d4a397/attachment.html>