[AstroPy] super-high-speed parsing of large FITS files

[David Kirkby]

Hi Stuart,

I was trying to use astropy.io.fits to read and write files with 45K HDUs a
while ago and eventually gave up and used the fitsio package, which was
much faster in this case.  You can see the relevant thread in the astropy
mailing list at
https://mail.scipy.org/pipermail/astropy/2015-January/thread.html

David

On Thu, Mar 2, 2017 at 3:07 PM Stuart P Littlefair <
s.littlefair at sheffield.ac.uk> wrote:

> Perhaps not the right place to ask this, but the official FITS mailing
> lists are overrun with spam.
>
> We are choosing a file format for a high speed photometer. One proposal is
> a MEF file. Frame rates are kHz, and there are five detectors so each FITS
> file may have many thousands of HDUs.
>
> One of our requirements is to be able to reduce the data in real time,
> whilst the data is being written. This means retrieving a given HDU from
> such a file in a msec or less. It would be OK if finding an initial HDU
> corresponding to a given frame number is slower than this,  if we can grab
> *subsequent* blocks of 5 HDUs (1 per detector) in < 1 msec.
>
> So I am wondering:
>
>  a) is this even possible in principle, with low-level C-code?
>  b) Can it be done via some un-sanctioned use of the private functions in
> the astropy FITs library?
>
> Many thanks in advance to anyone who can help
>
> --
> Stuart Littlefair
>
> -------------------------------------------------------
> Dept. of Physics & Astronomy,
> Univ. of Sheffield, Sheffield, S3 7RH.
>
> email: S.Littlefair at sheffield.ac.uk
> phone: +44 114 2224525 <+44%20114%20222%204525>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20170302/dfa2dd31/attachment.html>