[AstroPy] pyfits query: how to include a bit mask?

[Russell E Owen]

At 9:08 AM -0400 2005-10-11, Joe Harrington wrote:
>Remember that FITS is the flexible image TRANSPORT system.  Since your
>problem is data size, you can convert the data as you like, encode as
>FITS, store/transport, and re-convert into another FITS file.  The
>data are still self-documenting FITS even in their reduced state.  I'm
>not sure if this is any better than compressing the FITS images, but
>it might be, for archival purposes.
>
>Conversion ideas: pack the bits into bytes, use bitpix=8, and write
>software that unpacks it on the fly.  Or, since the flag image is
>likely to have a lot of zeros, do run-length encoding (what's used for
>faxes) on it and store it with any integer datatype you like.  Then it
>will store in practically no space.
>
>Sounds like what FITS needs is a BITPIX=1 extension, and some
>compression options.  Sounds like what FITS needs is to be HDF...

FITS could use a lot of things; it doesn't even support unsigned 
16-bit integers (despite the fact that the required combination of 
keywords is obvious -- it just happens to be disallowed -- and the 
fact that the vast majority of CCD cameras produce 16 bit unsigned 
integer data).

I wish we could use HDF. Astronomy is still pretty stuck on FITS. 
Maybe if one of the big new projects uses it (such as LSST), that'll 
break the logjam.

Packing the data is an interesting idea. I'm not sure I know how to 
do that in numarray but it's presumably doable.

Phil's suggestion of just putting both bitmasks in one 8 bit/pixel 
image is probably the best solution from a readability standpoint. A 
similar technique that I tried was to use a table of two logicals 
(one for each mask). If my calculations were right, it stored both of 
them together using 1 byte per pixel (just the same as using one 8 
bit image). It sounds like the main tradeoffs are that an image means 
ds9 can view it, whereas a table allows one to refer to each bitmask 
by name.

Long term, we should learn to transfer the data compressed. We 
transfer it via apache and it has the option to automatically 
compress "on the fly". The trick is determining IF the data was 
compressed and treating it appropriately. My code is written in 
python, but with the networking written in tcl, which makes this 
pretty tricky. I would like to eventually switch to Twisted Framework.

Thank you all for your help!

-- Russell