[AstroPy] question about fpack and astropy.io.fits

Erik Bray embray at stsci.edu
Thu Oct 9 16:24:28 EDT 2014


Hi Octavi,

On 10/08/2014 11:06 PM, Octavi Fors wrote:
> Hello everyone,
>
> this might sounds like a newbie question, but do you know if fpack compression
> package is supported by astropy.io.fits version 0.4.2?

It depends on what you mean, exactly.  It would help to add some clarification 
about what *fpack* is--apologies if this isn't new to you but it's worth 
mentioning for anyone else reading.

fpack itself (and its counterpart funpack (which I always enjoy reading "fun 
pack")) are just command-line programs that write new FITS files with one or 
more of the HDUs compressed according the FITS Tile Compression convention--or 
decompress them as the case may be.

In other words, the convention states how compressed FITS files should be read 
and written, while fpack is simply a program that knows how to write files with 
that convention.  There's no sense (I can think of) in which it makes sense for 
a general purpose FITS library (like astropy.io.fits) to support fpack itself. 
It does however support the same compression scheme and can read files 
compressed by fpack.  And likewise can write compressed files that can be read 
by funpack.

> In
> http://docs.astropy.org/en/stable/io/fits/usage/unfamiliar.html#compressed-image-data
> the module "astropy.io.fits.compression" is mentioned to access compressed HDUs,
> but it doesn't say if fpack is supported and how should this package be
> specified in fits.open method.

The docs you linked to at

http://astropy.readthedocs.org/en/stable/io/fits/usage/unfamiliar.html#compressed-image-data

could probably be expanded on with more examples, but as they indicate files 
containing compressed HDUs are handled transparently.  The compressed HDUs are 
accessed the same way as normal uncompressed IMAGE HDUs.  The data is 
automatically decompressed when it is read.  fpack has no role to play in that.


> My images images are 16 and -32bit. and I would like to make use of tiling
> pattern (FZTILE='(n,m)', see fpack manual
> <http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/docs/fpackguide.pdf>).
> For the -32bit images, any advise about noise-sensitive scaling (SCALE_FACTOR?)
> and subtractive dithering to achieve high compression ratios while preserving
> the scientific content of data, would be most than welcome.

The FZTILE keyword, as well as the other "Fpack compression directive keywords" 
listed in section 3.3 of the fpack guide are not standard FITS keywords and are 
only understood by the fpack program itself.

These can be used, for example, in a pipeline where raw data is written out 
uncompressed in a FITS file.  The file is later processed through fpack which 
will compress any HDUs marked with these keywords according to the indicated 
settings.  The end result of that is a new HDU compatible with the tile 
compression convention, and keywords like FZTILE become ZTILEn, or for example 
FZALGOR becomes ZCMPTYPE.

In other words, those FZ keywords indicate how the HDU *should* be compressed 
when passed through fpack.  The keywords on the actually compressed HDU 
(ZCMPTYPE, ZTILEn, etc.) indicate how the HDU *was* compressed.

The FZ* keywords have no effect in astropy.io.fits (though it might not be a bad 
idea to create a facility that understands them, for creating compressed HDUs 
from existing uncompressed HDUs).  Instead, to create a compressed HDU you just 
create a CompImageHDU object as demonstrated here:

http://astropy.readthedocs.org/en/stable/io/fits/usage/unfamiliar.html#creating-a-compressed-image-hdu

This works the same as other HDU types in the library.  Though it does take 
several optional keywords that are described in more detail in the API docs:

http://astropy.readthedocs.org/en/stable/io/fits/api/images.html#astropy.io.fits.CompImageHDU

For example to specify a tiling use

 >>> hdu = fits.CompImageHDU(data=data, header=header, tile_size=(n, m))

Subtractive dithering is used by default where applicable.  It otherwise 
supports all the features of the convention except perhaps the ZMASKCMP option 
(not for any particular technical reason--it's definitely something that can and 
should be done, though I haven't had any requests for it either...)

As to how to find a balance between compression quality and data quality I think 
others with more experience may wish to weigh in on that, though I will point 
out that documentation for the tiled image convention has some useful tips on that:

http://fits.gsfc.nasa.gov/registry/tilecompression/tilecompression2.3.pdf


Hope that helps,
Erik



More information about the AstroPy mailing list