[IPython-dev] roadmap for IPython.zmq.parallel
barrywark at gmail.com
Fri Jan 7 09:55:21 EST 2011
On Tue, Dec 21, 2010 at 2:04 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hey Barry,
> On Fri, Dec 17, 2010 at 9:15 PM, Barry Wark <barrywark at gmail.com> wrote:
>> It's been too long since I've been able to hang out in IPython land.
>> Given my previous interests, it's really exciting to see the work in
>> frontends accelerating with the new refactoring.
> Glad to have you around again!
>> I'm very excited to have a new opportunity to get back to IPython work
>> on a client project. The contract is to build a scientific data
>> processing and analysis framework. The analyses are expressed as a
>> DAG, with computation at the nodes done by exectuables that take a
>> standardized set of arguments and return a contracted output format.
>> Some of the executables are C, some Matlab, some Python, etc--standard
>> fare in academia. Our job is to build the engine to execute these
>> workflows, monitor results, etc. Jobs will initially execute on a
>> single machine (thus multiprocessing or a higher-level framework like
>> Rufus, http://www.ruffus.org.uk/) make sense, but the user may
>> eventually want to expand onto a local cluster.
> I'm still in India and will be offline as of tomorrow (traveling
> back)... But I'd suggest you have at least a look at:
> Nipype is Satra's brainchild (the same Satra who has committed the
> recent work on ipython) and it already has support for IPython's
> parallel execution using the 0.10.x code. It was also the motivation
> behind some of the new DAG support, as we hope in the future to have
> even better integration between nipype and ipython.
Yes, Nipype FTW! Awesome framework Satra!
> Satra is also in India right now (we were at the same conference) but
> he's on holiday for a fe days with his family and likely offline, so I
> expect him to pitch in only a little bit later. But I hope that in a
> few days when people's travel schedules normalize, we can see what can
> be done to benefit from common goals so that we reuse as much of the
> effort and manpower as possible.
> All the best,
More information about the IPython-dev