[IPython-dev] taskclient clear and hierarchical parallel processing

[Satrajit Ghosh]

hi brian and fernando,

Issue 1
---------
i create an ipython parallel cluster using ipcluster.

now shell 1:
launch python, get taskclient, run tasks with block=False

now shell 2:
launch python, get taskclient, run tasks with block=False
get my results using get_task_result
call taskclient.clear()

this clears all the tasks in shell 1.

is this the intended mode of operation?

alternatively, is there a way to clear a specific task based on its task id?
basically i want to use the same pool of resources from multiple client
connections.

Issue 2
---------
a related second question is more of a design pattern issue. i have
hierarchically complex DAGs such that each node of a DAG can be a DAG
itself. running this DAG in parallel leads me to the following issue.

let's assume i have 2 computational nodes on which i can send two nodes of
the DAG (Task 1 and Task 2). now suppose these tasks themselves are DAGs and
use the same mechanism for executing nodes. now when we push a task from
within Task1, these go into a queue and will never run (essentially wait for
the parent tasks to release the node).

now i can flatten out all the DAGs and run them, but it would be neat if
there was a pattern that would enable running these as concrete entities.

cheers,

satra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100512/f7b14f6e/attachment.html>