From rowen at u.washington.edu Fri Oct 7 14:56:29 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Fri, 7 Oct 2005 11:56:29 -0700 Subject: [AstroPy] pyfits query: how to include a bit mask? Message-ID: We're writing fits files with pyfits and would like to include a few bit masks (in particular "is saturated" and "ignore this pixel"). We're a bit stumped on how to do it and would appreciate any advice. -- Russell From chanley at stsci.edu Fri Oct 7 15:09:16 2005 From: chanley at stsci.edu (Christopher Hanley) Date: Fri, 07 Oct 2005 15:09:16 -0400 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: References: Message-ID: <4346C7DC.3030201@stsci.edu> Russell, My suggestion depends upon the type of FITS file you are creating. If you are dealing with FITS images, I would just add another extension to your FITS file containing the data quality flag values you wish to use. If you are writing FITS tables you could just add another column with the desired bit value. Chris -- Christopher Hanley Senior Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 Russell E Owen wrote: > We're writing fits files with pyfits and would like to include a few bit > masks (in particular "is saturated" and "ignore this pixel"). We're a > bit stumped on how to do it and would appreciate any advice. > > -- Russell > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.net > http://www.scipy.net/mailman/listinfo/astropy From rowen at u.washington.edu Mon Oct 10 19:37:32 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Mon, 10 Oct 2005 16:37:32 -0700 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: <4346C7DC.3030201@stsci.edu> References: <4346C7DC.3030201@stsci.edu> Message-ID: At 3:09 PM -0400 2005-10-07, Christopher Hanley wrote: >Russell, > >My suggestion depends upon the type of FITS file you are creating. >If you are dealing with FITS images, I would just add another >extension to your FITS file containing the data quality flag values >you wish to use. If you are writing FITS tables you could just add >another column with the desired bit value. Thank you for the helpful reply. These are simple fits images (one per fits file). If I add an extension then it will have to be at least 16 bits/pixel, right? The two masks I want to add are only 1 bit/pixel each (I could combine them into one, but they'd just have to be separated out again at the other end). Is there a natural and straightforward way to store the data in the fits file more efficiently? Transmission time is an issue (we could use compressed transmission, but that has its own headaches). -- Russell From shashi at prl.res.in Mon Oct 10 23:21:16 2005 From: shashi at prl.res.in (Shashikiran Ganesh) Date: Tue, 11 Oct 2005 08:51:16 +0530 (IST) Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: Message-ID: On Mon, 10 Oct 2005, Russell E Owen wrote: > If I add an extension then it will have to be at least 16 bits/pixel, > right? The two masks I want to add are only 1 bit/pixel each (I could > combine them into one, but they'd just have to be separated out again > at the other end). If your data are not using the entire 16bit/pixel range then you could use the highest two bits to store your flags. They would have to be extracted out at the other end though.. The other option would be to add a list of x and y locations with flag1 and another list corresponding to flag2 to the fits file. These are options only in the case where you want to minimize the size of the file. I think the previous suggestions are definitely more appropriate when size is not a constraint (easier to visualize with software like ds9 etc). --Shashi -- 2005-10-11 at 8:50am IST ------------------------------------------------------------------------------- Shashikiran Ganesh | shashi at prl.res.in Physical Research Laboratory | http://www.prl.res.in/~shashi Astronomy and Astrophysics Division | http://sarovar.org/users/shashi Ahmedabad 380 009, India | http://www.iap.fr/users/shashi ------------------------------------------------------------------------------- From hodge at stsci.edu Tue Oct 11 08:37:24 2005 From: hodge at stsci.edu (Phil Hodge) Date: Tue, 11 Oct 2005 08:37:24 -0400 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: References: <4346C7DC.3030201@stsci.edu> Message-ID: <434BB204.3020805@stsci.edu> Russell, > If I add an extension then it will have to be at least 16 bits/pixel, > right? The two masks I want to add are only 1 bit/pixel each (I could > combine them into one, but they'd just have to be separated out again > at the other end). No, a FITS image can be eight bits (unsigned) per pixel, BITPIX = 8. You could use a zero value to mean the pixel is good, 1 to mean "is saturated," and 2 to mean "ignore this pixel," for example. Since those are distinct bits in the pixel, they are easily separated, e.g. (dq & 1) is 1 if the pixel is saturated, and (dq & 2) is true if the pixel should be ignored. You would then be using 1/4 of the data volume in the data quality image. Phil From jh at oobleck.astro.cornell.edu Tue Oct 11 09:08:25 2005 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Tue, 11 Oct 2005 09:08:25 -0400 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: <434BB204.3020805@stsci.edu> (message from Phil Hodge on Tue, 11 Oct 2005 08:37:24 -0400) References: <4346C7DC.3030201@stsci.edu> <434BB204.3020805@stsci.edu> Message-ID: <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> Remember that FITS is the flexible image TRANSPORT system. Since your problem is data size, you can convert the data as you like, encode as FITS, store/transport, and re-convert into another FITS file. The data are still self-documenting FITS even in their reduced state. I'm not sure if this is any better than compressing the FITS images, but it might be, for archival purposes. Conversion ideas: pack the bits into bytes, use bitpix=8, and write software that unpacks it on the fly. Or, since the flag image is likely to have a lot of zeros, do run-length encoding (what's used for faxes) on it and store it with any integer datatype you like. Then it will store in practically no space. Sounds like what FITS needs is a BITPIX=1 extension, and some compression options. Sounds like what FITS needs is to be HDF... ;-) --jh-- From maurizio at iac.es Tue Oct 11 10:25:14 2005 From: maurizio at iac.es (Maurizio Panniello) Date: Tue, 11 Oct 2005 15:25:14 +0100 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> References: <4346C7DC.3030201@stsci.edu> <434BB204.3020805@stsci.edu> <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> Message-ID: <434BCB4A.50903@iac.es> An idea can be to store the masks as png files (you can use PIL library or directly with ImageMagik ) Png compress very well uniform patterns so if your mask have large uniform zone can reach very high compression factors. When you have trasfered the png files (included in fits as data if you want) you can than retransform these in fits format and join with your data for analisis. Joe Harrington wrote: >Remember that FITS is the flexible image TRANSPORT system. Since your >problem is data size, you can convert the data as you like, encode as >FITS, store/transport, and re-convert into another FITS file. The >data are still self-documenting FITS even in their reduced state. I'm >not sure if this is any better than compressing the FITS images, but >it might be, for archival purposes. > >Conversion ideas: pack the bits into bytes, use bitpix=8, and write >software that unpacks it on the fly. Or, since the flag image is >likely to have a lot of zeros, do run-length encoding (what's used for >faxes) on it and store it with any integer datatype you like. Then it >will store in practically no space. > >Sounds like what FITS needs is a BITPIX=1 extension, and some >compression options. Sounds like what FITS needs is to be HDF... >;-) > >--jh-- > >_______________________________________________ >AstroPy mailing list >AstroPy at scipy.net >http://www.scipy.net/mailman/listinfo/astropy > > From rowen at u.washington.edu Tue Oct 11 12:20:06 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Tue, 11 Oct 2005 09:20:06 -0700 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> References: <4346C7DC.3030201@stsci.edu> <434BB204.3020805@stsci.edu> <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> Message-ID: At 9:08 AM -0400 2005-10-11, Joe Harrington wrote: >Remember that FITS is the flexible image TRANSPORT system. Since your >problem is data size, you can convert the data as you like, encode as >FITS, store/transport, and re-convert into another FITS file. The >data are still self-documenting FITS even in their reduced state. I'm >not sure if this is any better than compressing the FITS images, but >it might be, for archival purposes. > >Conversion ideas: pack the bits into bytes, use bitpix=8, and write >software that unpacks it on the fly. Or, since the flag image is >likely to have a lot of zeros, do run-length encoding (what's used for >faxes) on it and store it with any integer datatype you like. Then it >will store in practically no space. > >Sounds like what FITS needs is a BITPIX=1 extension, and some >compression options. Sounds like what FITS needs is to be HDF... FITS could use a lot of things; it doesn't even support unsigned 16-bit integers (despite the fact that the required combination of keywords is obvious -- it just happens to be disallowed -- and the fact that the vast majority of CCD cameras produce 16 bit unsigned integer data). I wish we could use HDF. Astronomy is still pretty stuck on FITS. Maybe if one of the big new projects uses it (such as LSST), that'll break the logjam. Packing the data is an interesting idea. I'm not sure I know how to do that in numarray but it's presumably doable. Phil's suggestion of just putting both bitmasks in one 8 bit/pixel image is probably the best solution from a readability standpoint. A similar technique that I tried was to use a table of two logicals (one for each mask). If my calculations were right, it stored both of them together using 1 byte per pixel (just the same as using one 8 bit image). It sounds like the main tradeoffs are that an image means ds9 can view it, whereas a table allows one to refer to each bitmask by name. Long term, we should learn to transfer the data compressed. We transfer it via apache and it has the option to automatically compress "on the fly". The trick is determining IF the data was compressed and treating it appropriately. My code is written in python, but with the networking written in tcl, which makes this pretty tricky. I would like to eventually switch to Twisted Framework. Thank you all for your help! -- Russell From hodge at stsci.edu Tue Oct 11 13:39:13 2005 From: hodge at stsci.edu (Phil Hodge) Date: Tue, 11 Oct 2005 13:39:13 -0400 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: <200510111731.j9BHVf7J014783@oobleck.astro.cornell.edu> References: <4346C7DC.3030201@stsci.edu> <434BB204.3020805@stsci.edu> <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> <434BE840.3050801@stsci.edu> <200510111731.j9BHVf7J014783@oobleck.astro.cornell.edu> Message-ID: <434BF8C1.10209@stsci.edu> Joe, I apologize. I overreacted, and my message contributed nothing to the discussion. Phil From hodge at stsci.edu Tue Oct 11 13:52:26 2005 From: hodge at stsci.edu (Phil Hodge) Date: Tue, 11 Oct 2005 13:52:26 -0400 Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: References: <4346C7DC.3030201@stsci.edu> <434BB204.3020805@stsci.edu> <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> <434BE840.3050801@stsci.edu> <200510111731.j9BHVf7J014783@oobleck.astro.cornell.edu> <434BF8C1.10209@stsci.edu> Message-ID: <434BFBDA.2080507@stsci.edu> Russell, > At risk of jumping in, I must have missed something. ... Good! Actually, I didn't receive a copy of the message I posted either. I made a dumb remark about Joe Harrington's statement that FITS was the flexible image TRANSPORT system. Phil From jpb at pha.jhu.edu Tue Oct 11 17:19:57 2005 From: jpb at pha.jhu.edu (John Blakeslee) Date: Tue, 11 Oct 2005 17:19:57 -0400 (EDT) Subject: [AstroPy] pyfits query: how to include a bit mask? In-Reply-To: References: <4346C7DC.3030201@stsci.edu> <434BB204.3020805@stsci.edu> <200510111308.j9BD8P91013741@oobleck.astro.cornell.edu> Message-ID: Hi all, Just a minor note on this - >> Sounds like what FITS needs is a BITPIX=1 extension, There is sort of an 'informal standard' for this, that's been around for quite a while. John Tonry distributes a version of Vista that allows BITPIX=1 (FITS bitmap), which is used for mask images. It's not quite as space efficient as the iraf pixel list representation, but it's nice because it's an image that you can display and pan around (though things like ds9 choke on it). He distributes the code at - http://www.ifa.hawaii.edu/~jt/soft.html and the relevant source file is called fitsdisk.c (which also defines a format for unsigned integer FITS, of course). It's easy to compile it into other programs if you want them to read this FITS bitmap format. Ultimately, it would be nice to integrate this economical standard into numarray, but this is the stuff that dreams are made of.... John