Francesc Alted falted@pytables.org writes:
Well, if you are pondering using parallel reading because of speed,
I was actually pondering using parallel _writing_ because of speed. Parallel reading is easy: each process just opens the file and reads independently. But merely switching to NetCDF gave a decent speed improvement even with sequential writing.
Are you sure? Here you have a couple of OpenDX data importers for HDF5:
I was aware of dxhdf5 but I don't think it handles irregular meshes. It seems that the Cactus one doesn't either.
Before doing that, talk with Konrad. I know that Scientific Python supports MPI and BSPlib right-out-of-the-box, so maybe there is a shorter path to do what you want.
Unfortunately I was not able to use Konrad's MPI bindings. Petsc has its own initialization routine that needs to be called early on. I had to create another special version of the Python interpreter, different from Konrad's. I also needed more functionality than Konrad's bindings have - I even use MPI_Alltoallv at one point. Fortunately creating my own MPI bindings with Swig and Numarray was fairly easy.
So, perhaps spending your time writing Python bindings for Parallel-HDF5 would be a better bet for future applications.
Perhaps, but first I'll have to concentrate on the actual number crunching code to get some data to write. Then I'll see whether I really need parallel writing.
Thanks to everybody for helpful suggestions.