Assignment from a list is slow in Numarray
Hello,
I am new to the list, sorry if you've been through this before.
I am trying to do some FEM computations using Petsc, to which I have written Python bindings using Swig. That involves passing arrays around, which I found delightfully simple with NA_{Input,Output,Io}Array. Numeric seems more difficult for output and bidirectional arrays.
My code for reading a triangulation from a file went roughly like this:
coord = zeros( (n_vertices, 2), Float) for v in n_vertices: coord[ v, :] = [float( s) for s in file.readline().split()]
This was taking quite a bit of time with ~50000 vertices and ~100000 elements, for which three integers per element are read in a similar manner. I found it was faster to loop explicitly:
coord = zeros( (n_vertices, 2), Float) for v in n_vertices: for j, c in enumerate( [float( s) for s in file.readline().split()]): coord[ v, j] = c
Morally this uglier code with an explicit loop should not be faster but it is with Numarray. With Numeric assignment from a list has reasonable performance. How can it be improved for Numarray?
A Diumenge 19 Setembre 2004 19:35, Timo Korvola va escriure:
My code for reading a triangulation from a file went roughly like this:
coord = zeros( (n_vertices, 2), Float) for v in n_vertices: coord[ v, :] = [float( s) for s in file.readline().split()]
This was taking quite a bit of time with ~50000 vertices and ~100000 elements, for which three integers per element are read in a similar manner. I found it was faster to loop explicitly:
coord = zeros( (n_vertices, 2), Float) for v in n_vertices: for j, c in enumerate( [float( s) for s in file.readline().split()]): coord[ v, j] = c
Morally this uglier code with an explicit loop should not be faster but it is with Numarray. With Numeric assignment from a list has reasonable performance. How can it be improved for Numarray?
If you want to achieve fast I/O with both numarray/Numeric, you may want to try PyTables (wwww.pytables.org). It supports numarray objects natively, so you should get pretty fast performance. At the beginning, you will need to export your data to a PyTables file, but then you can read data as many times as you want from it.
HTH
Francesc Alted falted@pytables.org writes:
At the beginning, you will need to export your data to a PyTables file,
... which appears to be actually a HDF5 file. Thanks for the tip. It is clear that a binary file format would be more advantageous simply because text files are not seekable in the way needed for parallel reading. I was thinking of using NetCDF because OpenDX does not support HDF5. Konrad Hinsen has written a Python interface for reading NetCDF files. Distributed writing is more compilcated and unfortunately this interface seems particularly unsuitable for it because the difference between definition and data mode is hidden. The interface also uses Numeric instead of Numarray.
An advantage of HDF5 would be that the libraries support parallel I/O via MPI-IO but can this be utilised in PyTables? There is the problem that there are no standard MPI bindings for Python.
I have also considered writing Python bindings for Parallel-NetCDF but I suppose that would not be totally trivial even if the library turns out to be well Swiggable.
A Dilluns 20 Setembre 2004 15:16, Timo Korvola va escriure:
... which appears to be actually a HDF5 file. Thanks for the tip. It is clear that a binary file format would be more advantageous simply because text files are not seekable in the way needed for parallel reading.
Well, if you are pondering using parallel reading because of speed, try first PyTables, you may get surprised how fast it can be. For example, using the same example that Todd has sent today (i.e. writing and reading an array of (10**5,3) integer elements), I've re-run it using PyTables and, just for the sake of comparison, NetCDF (using the Scientific Python wrapper). Here are the results (using a laptop with Pentium IV @ 2 GHz with Debian GNU/Linux):
Time to write file (text mode) 2.12 sec Time to write file (NetCDF version) 0.0587 sec Time to write file (PyTables version) 0.00682 sec Time to read file (strings.fasteval version) 0.259 sec Time to read file (NetCDF version) 0.0470 sec Time to read file (PyTables version) 0.00423 sec
so, for reading, PyTables can be more than 60 times faster than numarray.strings.eval and almost 10 times faster than Scientific.IO.NetCDF (the latter using Numeric). And I'm pretty sure that these ratios would increase for bigger datasets.
I was thinking of using NetCDF because OpenDX does not support HDF5.
Are you sure? Here you have a couple of OpenDX data importers for HDF5:
http://www.cactuscode.org/VizTools/OpenDX.html http://www-beams.colorado.edu/dxhdf5/
An advantage of HDF5 would be that the libraries support parallel I/O via MPI-IO but can this be utilised in PyTables? There is the problem that there are no standard MPI bindings for Python.
Curiously enough Paul Dubois asked me the very same question during the recent SciPy '04 Conference. And the answer is the same: PyTables does not support MPI-IO at this time, because I guess that could be a formidable developer time waster. I think I should try first make PyTables threading-aware before embarking myself in larger entreprises. I recognize, though, that a MPI-IO-aware PyTables would be quite nice.
I have also considered writing Python bindings for Parallel-NetCDF but I suppose that would not be totally trivial even if the library turns out to be well Swiggable.
Before doing that, talk with Konrad. I know that Scientific Python supports MPI and BSPlib right-out-of-the-box, so maybe there is a shorter path to do what you want.
In addition, you must be aware that the next version of NetCDF (the 4), will be implemented on top of HDF5 [1]. So, perhaps spending your time writing Python bindings for Parallel-HDF5 would be a better bet for future applications.
[1] http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4/index.html
Cheers,
Francesc Alted falted@pytables.org writes:
Well, if you are pondering using parallel reading because of speed,
I was actually pondering using parallel _writing_ because of speed. Parallel reading is easy: each process just opens the file and reads independently. But merely switching to NetCDF gave a decent speed improvement even with sequential writing.
Are you sure? Here you have a couple of OpenDX data importers for HDF5:
I was aware of dxhdf5 but I don't think it handles irregular meshes. It seems that the Cactus one doesn't either.
Before doing that, talk with Konrad. I know that Scientific Python supports MPI and BSPlib right-out-of-the-box, so maybe there is a shorter path to do what you want.
Unfortunately I was not able to use Konrad's MPI bindings. Petsc has its own initialization routine that needs to be called early on. I had to create another special version of the Python interpreter, different from Konrad's. I also needed more functionality than Konrad's bindings have - I even use MPI_Alltoallv at one point. Fortunately creating my own MPI bindings with Swig and Numarray was fairly easy.
So, perhaps spending your time writing Python bindings for Parallel-HDF5 would be a better bet for future applications.
Perhaps, but first I'll have to concentrate on the actual number crunching code to get some data to write. Then I'll see whether I really need parallel writing.
Thanks to everybody for helpful suggestions.
Here's another possible I/O approach which uses numarray.strings to create and evaluate a CharArray using the new for 1.1 method, fasteval(). This method is dependent on fixed length fields of data and doesn't currently handle 64-bit integer types or complex numbers the way I'd like, but it may be useful for your case.
I created my test file like this:
for i in range(10**5):
... f.write("%10d %10d %10d\n" % (i, i, i)) ...
Then I created a CharArray from the file like this:
import numarray.strings as str f = open("test.dat", "r") c = str.fromfile(f, shape=(10**5, 3), itemsize = 11) c
CharArray([[' 0', ' 0', ' 0'], [' 1', ' 1', ' 1'], [' 2', ' 2', ' 2'], ..., [' 99997', ' 99997', ' 99997'], [' 99998', ' 99998', ' 99998'], [' 99999', ' 99999', ' 99999']])
Finally, I converted it into a NumArray, with reasonable performance, like this:
n = c.fasteval(type=Int32) n
array([[ 0, 0, 0], [ 1, 1, 1], [ 2, 2, 2], ..., [99997, 99997, 99997], [99998, 99998, 99998], [99999, 99999, 99999]])
Hope this helps, Todd
participants (3)
-
Francesc Alted -
Timo Korvola -
Todd Miller