[SciPy-User] tool for running simulations

[Gael Varoquaux]

On Fri, Jun 24, 2011 at 01:22:53AM +0200, Dan Goodman wrote:
> >> * Can it be used on multiple computers?

> > If you have an NFS share between the computers, yes. The code works OK in
> > parallel. You will have race conditions, but it captures them, and falls
> > back on its feets.

> Ah nice, how does it do that?

Breaking up the storage in many different files, located in directories,
and enforcing consistency only localy.

> >> * Can you browse the generated data easily?

> > No. This is something that could/should be improved (want to organize a
> > sprint in Paris, if you still are in Paris?).

> If we used HDF5 as the backend then you'd get this for free, so maybe 
> that's the better way?

I wouldn't call it a 'better way'. It is a different way that brings in
some good things. I do not think that HDF5 can be as robust as my current
implementation to crashes. I addition, it enforces a bug depency. I want
joblib to work with no dependencies on Python. It can have optional
dependencies though.

> I'd possibly be interested in doing a sprint, I'm in Paris until the
> end of July but I'm finishing up here so I'll probably have quite a lot
> of things to finish.

OK. I am abroad till the end of July :).

G