[IPython-dev] roadmap for IPython.zmq.parallel

[Barry Wark]

Hi all,

It's been too long since I've been able to hang out in IPython land.
Given my previous interests, it's really exciting to see the work in
frontends accelerating with the new refactoring.

I'm very excited to have a new opportunity to get back to IPython work
on a client project. The contract is to build a scientific data
processing and analysis framework. The analyses are expressed as a
DAG, with computation at the nodes done by exectuables that take a
standardized set of arguments and return a contracted output format.
Some of the executables are C, some Matlab, some Python, etc--standard
fare in academia. Our job is to build the engine to execute these
workflows, monitor results, etc. Jobs will initially execute on a
single machine (thus multiprocessing or a higher-level framework like
Rufus, http://www.ruffus.org.uk/) make sense, but the user may
eventually want to expand onto a local cluster.

MinRK's IPython.zmq.parallel branch, with its support for DAG
dependencies looks like it might fit the bill as a base for our work.
I'm curious what you think is the status and timeline of this branch.
I am happy to dedicate time to improving and helping with the
IPython.zmq.parallel branch; the contract includes 1/4 time for the
duration of the project for work on project dependencies. The timeline
for deploying our project is roughly Feb-March. Is it
reasonable/adviseable to build on IPython.zmp.parallel in that
timeframe? It looks like ssh tunnels are the current basis for
security in the zmq branch. Is that correct? Are there any plans to
implement any sort of pluggable authentication/authorization?

Thanks,
Barry Wark