[Numpy-discussion] Assignment from a list is slow in Numarray

[Timo Korvola]

Francesc Alted <falted at pytables.org> writes:
> Well, if you are pondering using parallel reading because of speed,

I was actually pondering using parallel _writing_ because of speed.
Parallel reading is easy: each process just opens the file and reads
independently.  But merely switching to NetCDF gave a decent speed
improvement even with sequential writing.

> Are you sure? Here you have a couple of OpenDX data importers for HDF5:

I was aware of dxhdf5 but I don't think it handles irregular meshes.
It seems that the Cactus one doesn't either.

> Before doing that, talk with Konrad. I know that Scientific Python supports
> MPI and BSPlib right-out-of-the-box, so maybe there is a shorter path to do
> what you want.

Unfortunately I was not able to use Konrad's MPI bindings.  Petsc has
its own initialization routine that needs to be called early on.  I
had to create another special version of the Python interpreter,
different from Konrad's.  I also needed more functionality than
Konrad's bindings have - I even use MPI_Alltoallv at one point.
Fortunately creating my own MPI bindings with Swig and Numarray was
fairly easy.

> So, perhaps spending your time writing Python bindings for
> Parallel-HDF5 would be a better bet for future applications.

Perhaps, but first I'll have to concentrate on the actual
number crunching code to get some data to write.  Then I'll see
whether I really need parallel writing.

Thanks to everybody for helpful suggestions.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>