[SciPy-User] tool for running simulations

[David Baddeley]

I'm also of the opinion that an hdf backend might not solve your problems - with 
my system I've resorted to using a single server which arbitrates and abstracts 
reading and writing hdf along with a whole pile of locking to get around race 
conditions wrt. reading and writing HDF. It's OK if computation >> data flow, 
but can become problematic if this isn't the case. I'm accessing HDF through 
pytables and there's also a problem with the windows version of the hdf 
libraries that are shipped with pytables not being threadsafe - meaning that I 
have to take out a global lock on the library even when I'm accessing two 
different files. Am still trying to find the time to polish up a few comments so 
I can give you the code in a somewhat readable state.

cheers,
David


----- Original Message ----
From: Dan Goodman <dg.gmane at thesamovar.net>
To: scipy-user at scipy.org
Sent: Sat, 25 June, 2011 10:48:18 PM
Subject: Re: [SciPy-User] tool for running simulations

On 25/06/2011 00:49, Gael Varoquaux wrote:
> On Fri, Jun 24, 2011 at 01:22:53AM +0200, Dan Goodman wrote:
>>>> * Can it be used on multiple computers?
>
>>> If you have an NFS share between the computers, yes. The code works OK in
>>> parallel. You will have race conditions, but it captures them, and falls
>>> back on its feets.
>
>> Ah nice, how does it do that?
>
> Breaking up the storage in many different files, located in directories,
> and enforcing consistency only localy.

Ah yes, so I'm using a similar system, and indeed it won't extend well 
to working with HDF5 as the backend.

>>>> * Can you browse the generated data easily?
>
>>> No. This is something that could/should be improved (want to organize a
>>> sprint in Paris, if you still are in Paris?).
>
>> If we used HDF5 as the backend then you'd get this for free, so maybe
>> that's the better way?
>
> I wouldn't call it a 'better way'. It is a different way that brings in
> some good things. I do not think that HDF5 can be as robust as my current
> implementation to crashes. I addition, it enforces a bug depency. I want
> joblib to work with no dependencies on Python. It can have optional
> dependencies though.

Good points.

>> I'd possibly be interested in doing a sprint, I'm in Paris until the
>> end of July but I'm finishing up here so I'll probably have quite a lot
>> of things to finish.
>
> OK. I am abroad till the end of July :).

Ah OK, so maybe some time after September then? Unless you'll be in 
London in August? :)

Dan

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user