[SciPy-User] tool for running simulations

[Dan Goodman]

>> At the moment I have a system built on Python shelves, and the
>> performance is not great.
>
> :). On top of that, I had a fair amount of database corruptions when I
> was using shelves. This code is reasonnably isolated, so you won't
> corrupt your complete cache, just one result.

Yes, I'm having to make lots of backups because a single interrupted 
write can completely destroy the whole database it seems. (Sometimes I 
can rescue some of the data, sometimes not.)

>> * Can it be used on multiple computers?
>
> If you have an NFS share between the computers, yes. The code works OK in
> parallel. You will have race conditions, but it captures them, and falls
> back on its feets.

Ah nice, how does it do that?

>> * Can you browse the generated data easily?
>
> No. This is something that could/should be improved (want to organize a
> sprint in Paris, if you still are in Paris?).

If we used HDF5 as the backend then you'd get this for free, so maybe 
that's the better way? I'd possibly be interested in doing a sprint, I'm 
in Paris until the end of July but I'm finishing up here so I'll 
probably have quite a lot of things to finish.

>> That's one thing I liked about the idea of doing it with HDF5 is that
>> there are nice visual browsers and you can include metadata, search via
>> metadata, remove parts of the data, etc.
>
> Agreed. Actually, an HDF5 backend would probably be a good idea. But
> first we would need to merge Dags's changes, that abstract a bit the data
> storage.

I like the idea of putting it in HDF5 because it's becoming quite a good 
standard for scientific computing, so potentially this makes it easier 
to integrate with other tools, etc., without having to write translators.

I'm not in a huge hurry to get this working because I'm too deep into my 
current project to switch from using shelves - annoying as they are. 
But, I really don't want to have to go through all that hassle again so 
I'm definitely motivated to do something in the relatively near future. 
I'm hoping to be in London on a grant that will be taking me back to 
Paris reasonably often, so we could arrange some time next (academic) 
year if not before.

Dan