[AstroPy] reading one line from many small fits files

[Derek Homeier]

Hi Erik,

On 31.07.2012, at 5:52PM, Erik Bray <embray at stsci.edu> wrote:

> It probably shouldn't matter whether or not he's calling close(), but 
> the question about BSCALE/BZERO is possibly relevant.  Is the data 
> you're reading from an image or a table?  If it's an image, then as 
> Derek wrote PyFITS is still pretty inefficient, in that it will 
> transform the entire image, even if using mmap (which is the default now 
> by the way). I have plans for overhauling this but it hasn't been a high 
> priority for the most part.  You can also turn off image scaling if you 
> use do_not_scale_image_data=True when opening the file.  That might 
> speed things up.
> 
> This is one area where using the .section feature on Image HDUs might 
> still be useful.  For example:
> 
> result[i] = photo[1].section[x[otherfield] - 1]
> 
> In PyFITS 3.1 that has improved support for scaling just sections of the 
> file, which didn't work well before.  That might also be faster.
> 
> Of course this is all a moot point if this is not a scaled image.  In 
> any case, opening a file and reading a single row out of the data should 
> not generally take as long as 1 second--especially if they're small files.

indeed the question was about tables, I just still had the discussion about automatic 
scaling and the potential type conversion coming with it in mind. And the docs mention

`FITS_rec` is the data part of a table HDU's data part.  This is a
layer over the `recarray`, so we can deal with scaled columns.

so I was wondering if some automatic scaling features might have been added 
(or be in testing) in 3.1 svn.

Cheers,
						Derek