[IPython-dev] roadmap for IPython.zmq.parallel
fperez.net at gmail.com
Tue Dec 21 02:04:59 EST 2010
On Fri, Dec 17, 2010 at 9:15 PM, Barry Wark <barrywark at gmail.com> wrote:
> It's been too long since I've been able to hang out in IPython land.
> Given my previous interests, it's really exciting to see the work in
> frontends accelerating with the new refactoring.
Glad to have you around again!
> I'm very excited to have a new opportunity to get back to IPython work
> on a client project. The contract is to build a scientific data
> processing and analysis framework. The analyses are expressed as a
> DAG, with computation at the nodes done by exectuables that take a
> standardized set of arguments and return a contracted output format.
> Some of the executables are C, some Matlab, some Python, etc--standard
> fare in academia. Our job is to build the engine to execute these
> workflows, monitor results, etc. Jobs will initially execute on a
> single machine (thus multiprocessing or a higher-level framework like
> Rufus, http://www.ruffus.org.uk/) make sense, but the user may
> eventually want to expand onto a local cluster.
I'm still in India and will be offline as of tomorrow (traveling
back)... But I'd suggest you have at least a look at:
Nipype is Satra's brainchild (the same Satra who has committed the
recent work on ipython) and it already has support for IPython's
parallel execution using the 0.10.x code. It was also the motivation
behind some of the new DAG support, as we hope in the future to have
even better integration between nipype and ipython.
Satra is also in India right now (we were at the same conference) but
he's on holiday for a fe days with his family and likely offline, so I
expect him to pitch in only a little bit later. But I hope that in a
few days when people's travel schedules normalize, we can see what can
be done to benefit from common goals so that we reuse as much of the
effort and manpower as possible.
All the best,
More information about the IPython-dev