[AstroPy] PyFITS 0.6.2 available

[Perry Greenfield]

<x-charset Windows-1252>Clive Page writes

> Thanks for your very instructive posting.  I now fully understand your
> motivation.  I have been happily using CFITSIO for many years and have
> made some attempts to put a higher level interface on it, but with limited
> success.  But you face the problem that CFITSIO already has support for a
> host of features that people already use in their C and Fortran programs,
> and will expect in PyFITS.  For example:
>
> - scaling using BZERO/BSCALE in images and tables (I note that's on your
> list of things to do soon)
>
> - full support for null values in all data types (and IEEE special values
> in floating-point cases)
>
These will be supported as well.

> - support for vector columns and variable length arrays in BIN tables
>
I believe vector columns are already supported. Variable length arrays
will be as well (though efficiency is an issue since array objects must
be created for each variable length entry.

> - almost transparent access to ASCII and BINARY tables (I will I didn't
> have to put that "almost" there).
>
I'm not sure what you mean by that. Do you mean not having to worry
about whether it is of one type or the other?

> - support for long string keywords (comments flowing over several lines)
>
Not sure what we will do about this.

> - support for units in keyword comments
>
Or this. I imagine we can add this if there is demand.

> - support for arrays of strings (rA:SSTRw/nn convention and an earlier
> one)
>
We will have to look at this.

> - syntax to filter tables from command-line using row expressions etc
>
We will certainly add functionality like this.

> - access to FITS files over the net using FTP and HTTP
>
This shouldn't be hard to add.

> - automatic data type conversion in many cases
>
I'm not sure this applies to a Python interface. I'd need to
see a more specific request of what automatic conversion behavior
was desired. There is the issue of whether types are converted for
keyword values, table columns, or image data and in what context.

> - support for HIERARCH keyword convention.
>
Not sure we would support this. I think there are much better ways
of doing this. But if it is heavily demanded we might consider it.

> - support for automatic decompression of compressed files
>
Compressed at the file level or extension level?
This is something we do want to support (we were invovled in the
definition the extension compression conventions after all :-).
Decompressing at the file level should be fairly easy to support.

> Now I have only ever used about half of these, but I suspect other FITS
> programmers have used a different subset, so PyFITS will need to support
> rather a lot to keep the majority of users happy.  That could be quite a
> lot of work.
>
A FITS library is a lot of work :-) This is unavoidable by its nature.
We were aware of that when we started.

> > 3) It is not possible to memory map FITS files in CFITSIO.
>
> I can see that memory mapping is desirable in many areas.  But I have some
> reservations.  I have past experience of using the Starlink HDS library,
> which relied on memory mapping.  But there is a downside if you run out of
> physical memory - the system reads your file and then pages it to disc,
> usually another disc.  So each logical read turns into a physical read, a

Memory mapping is not a panacea for memory issues. It does make it easier
to manage memory and can provide a means for significant improvements
in accessing large data sets. But I don't think it means one can forget
about the fact that one is dealing with large data sets--far from it.

It is particularly useful if one only needs to access a subset of the
data (thus most of it is never read into memory). In a similar vein, we
are going to see if it provides the basic mechanism for reading data
a subsection at a time (memory map it and then use arrau slicing to
access the subsets rather than do explicit I/O on array subsets).
If that proves impractical, we can always fall back to providing methods
to read subsets of arrays (it's not as nice an interface though).
But I agree, if you memory map large arrays and do anything that results
in copies of the array being generated (e.g., from ufuncs) you
will run out of virtual memory or page heavily. On the other hand, if
you perform "compressive" operations (doing statistics on the image or
reduction operations) then memory mapping can be a big win.

> write, and another read, and efficiency drops catastrophically.  There is
> another problem: here in the XMM-Newton project we use a FITS
> infrastructure (built in top of CFITSIO) which always reads each input
> file to memory before doing anything with it.  Just last month we came
> across our first FITS file which exceeded the memory addressing limits of
> Solaris 2.6 (2 GB I think); our only solution may be to move to a 64-bit
> verison of the operating system.
>
This is a problem in its own right!

Anyway, please do provide us feedback.

Thanks, Perry

_____________________________________________________
AstroPy mailing list  -  astropy at stsci.edu
http://lheawww.gsfc.nasa.gov/~bridgman/AstroPy/

</x-charset>