[AstroPy] Streaming binary data format for spectra

Rick Wagner rwagner at physics.ucsd.edu
Fri Apr 25 00:25:03 EDT 2008


> I am working on radio astronomy instrumentation. In particular I have
> a spectrometer which dumps spectra very rapidly. I am using Python to
> capture this data and plot this data. In evaluating the equipment, I
> would like to make long running Allan variance measurements, and thus
> I would like to save every spectrum to disk for post processing. I
> realize I could dump the data to disk as just raw binary data, but if
> possible I would like to use an existing data format to avoid
> reinventing the wheel for reading and writing operations. I prefer to
> not aggregate everything into RAM and then dump it all do disk at
> once; it would be better to stream it to disk.
> I looked into using FITS, but it doesn't seem suited to dynamically
> increasing the size of the file. I tried using the pyfits streaming
> HDU construct, but it seemed to require specifying an initial dataset
> size first, then filling that dataset.
> NetCDF seemed like a good solution since it allows for an unlimited
> dimension to your data set, but I have not had any luck getting any of
> the python wrappers to work with this library.
> Does anyone have any suggestions of a data format that would fit my  
> needs?

I would recommend using the library that NetCDF is built on top of,  
HDF5 [1]. The Pytables [2] code provides a nice interface to the  
data, and you can get to your data from other languages, like C, C++,  
Fortran, etc. We use HDF5 in our group for massively (1000+  
processor) simulations, and find it very fast, and the HDF5 is great  
to work with.

If you have any questions, I'd be glad to help you get started.


[1] http://hdfgroup.org/
[2] http://www.pytables.org/

Rick Wagner, Graduate Student Researcher
UCSD Physics
9500 Gilman Drive
La Jolla, CA  92093-0424
Email:  rwagner at physics.ucsd.edu
WWW:    http://lca.ucsd.edu/projects/rpwagner
(858) 822-4784 Phone
Measuring programming progress by lines of code is
like measuring aircraft building progress by weight.
--Bill Gates

More information about the AstroPy mailing list