[AstroPy] Moving Ahead with Raw Image Conversion

Tue Apr 14 13:40:13 EDT 2009

Yes, that's true, but I was trying to express the confusion over leaving 
the pyfits manual lingering in numarray-land.

Jim Vickroy wrote:
> At the risk of stating the obvious, you should only be using numpy 
> (not numarray).  Numarray was a competing package that pyfits 
> previously supported.  Numarray is deprecated and the recent releases 
> of pyfits have complete support for numpy.
>
> -- jv
>
> Wayne Watson wrote:
>> Howdy, I think I see the ccdsoft problem. I looked at a fit (fits) file 
>> I produced with my CCD camera using it, and it produces a uint16 
>> (BITPIX=16). I just made the change you suggested, and the resulting 
>> file shows BITPIX=8, but CCDSoft liked it. Apparently, it objected to 
>> the 32 bit image earlier, which probably gave rise to the black image. 
>> The former way produced a file of 1,204KB, and the current file produced 
>> sized with the simple replacement produced a file of 304KB. I used 
>> dtype='uint8' instead of 'B'.
>>
>> Somewhere along the line I confused myself with the numarray use in the 
>> FITS manual, and apparently its look-alike notion ( as seen via arange 
>> and reshape) in numpy. Maybe they are equivalent in every respect. I was 
>> trying to figure out how to get Python to put together a 'matrix' 
>> (mathematical sense) to feed it to data, but it looks like reshape is 
>> the ticket. reshape seems missing from numarray (not mentioned in the 
>> fits manual), so maybe there is a difference. I'm not sure the pyfits 
>> manual ever showed how to make a 'matrix' and stuff it into the data 
>> portion of a FITS header (maybe that's what section 2.2.3 is about, 
>> which interestingly has the same title as 2.2.2).
>>
>> I think we are pretty much, successfully, done here. Allowing the meteor 
>> program I'm adding new features to be able to use FITS will be a very 
>> useful change. Some exposure to pyfits and numpy helps too from the 
>> coding aspects. Of course, if I decide to add features that allow FITS 
>> files into the program, say, for IP (img processing) work, that'll 
>> require some unraveling of the data part of the fits file for 
>> PIL/Tinter. Later though.
>>
>> Thanks for the help.
>>
>> Peter Erwin wrote:
>>   
>>> Hi Wayne,
>>>
>>> Anne Archibald pointed out a much more concise and elegant way to read in
>>> and convert the raw file, using a function in Numpy.  So you could 
>>> replace the
>>> following original code:
>>>
>>>     
>>>> # get raw data from .raw file:
>>>> raw_file=open('sent_internal.raw','rb')
>>>> raw_image=raw_file.read()
>>>> raw_file.close()
>>>>
>>>> # convert raw data to Python list of integers (1-D):
>>>> rawim_intarray = []
>>>> for x in raw_image:
>>>>   # convert byte to integer ('B' = treat data as single unsigned byte)
>>>>   newint = struct.unpack('B', x)[0]
>>>>   rawim_intarray.append(newint)
>>>>
>>>> # convert Python list of integers into numpy array of integers (note: 
>>>> still 1-D at this point)
>>>> rawim_numpy = numpy.array(rawim_intarray)
>>>>       
>>> with this:
>>>
>>> rawim_numpy = numpy.fromfile('sent_internal.raw', dtype='B')
>>>
>>> [note: dtype='B' is identical to dtype=numpy.uint8, which Anne used in 
>>> her example]
>>>
>>> and then continue as before.
>>>
>>> This has the added benefit of producing a smaller output file (300 KB vs
>>> 1.2 MB), since the numpy array is now explicitly in 8-bit-integer form
>>> (rather than the 32-bit-integer form that my code created), and so
>>> Pyfits will automatically save it as 8-bit integers (you can check this
>>> by looking at the header keyword BITPIX).
>>>
>>>
>>> You're right, the raw image is a sequence of (standard) 8-bit bytes.  
>>> Well, all
>>> computer files are streams of bytes; the question is how to interpret 
>>> them.  Since
>>> you said earlier that it was supposed to be 8-bit values, the idea is 
>>> then to
>>> interpret each individual byte as an unsigned "integer" value (values 
>>> from 0 to 255);
>>> that's what the "dtype='B'" (or "dtype=numpy.uint8") option passed to 
>>> numpy.fromfile()
>>> does.  (My previous code did the same thing, but then assigned each 
>>> value to a
>>> standard Python integer, which is 4 bytes long; perfectly workable, 
>>> but a bit
>>> wasteful!).
>>>
>>> Anyway, when I do this, I can see an image in DS9 (a circular aperture 
>>> with ragged
>>> edges and an overall horizontal gradient inside the aperture).  So I'm 
>>> pretty sure
>>> it's working.  (I've attached a jpeg saved from DS9 to show what the 
>>> image looks like.)
>>>
>>> I don't think I can help you with this "ccdsoft" program; I'm not 
>>> familiar with it.
>>> If it's a Windows program, then it may expect all FITS files to end in 
>>> ".fit" instead
>>> of ".fits", just because Windows traditionally expects files to have 
>>> 3-letter extensions.
>>>
>>>
>>> As for overwriting an existing FITS file, try this:
>>>
>>> hdu.writeto("existing_file_name.fit", clobber=True)
>>>
>>> cheers,
>>>
>>> Peter
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>>
>>>
>>> On Apr 14, 2009, at 6:28 AM, Wayne Watson wrote:
>>>
>>>     
>>>> Hi, Peter. It looks like I confused a few people with the hex. I had 
>>>> removed it, but only after I placed the code in the msg. I was 
>>>> playing with hex in the event I might need it somehow. int was fine 
>>>> for my experimental purposes.
>>>>
>>>> Here's your code with a minor correction. I changed this line:
>>>>  rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>>> The difference is rawmin_numpy.
>>>> I added a print at the end, and threw in two lines that attempt to 
>>>> put header info into the final file.  Continuing below the code.
>>>>
>>>> =====================Start===============
>>>> import struct, numpy, pyfits   # struct is part of the main Python 
>>>> library
>>>>
>>>> # get raw data from .raw file:
>>>> raw_file=open('sent_internal.raw','rb')
>>>> raw_image=raw_file.read()
>>>> raw_file.close()
>>>>
>>>> # convert raw data to Python list of integers (1-D):
>>>> rawim_intarray = []
>>>> for x in raw_image:
>>>>   # convert byte to integer ('B' = treat data as single unsigned byte)
>>>>   newint = struct.unpack('B', x)[0]
>>>>   rawim_intarray.append(newint)
>>>>
>>>> # convert Python list of integers into numpy array of integers (note: 
>>>> still 1-D at this point)
>>>> rawim_numpy = numpy.array(rawim_intarray)
>>>>
>>>> # reshape numpy array into 2D form, using our knowledge of the 
>>>> original image's
>>>> # x and y sizes (kudos to Megan Sosey for pointing out how to do this)
>>>> rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>>>
>>>> # create Pyfits header-data unit:
>>>> hdu = pyfits.PrimaryHDU(rawim_numpy_2d)
>>>> hdu.header.update('LATOBS',"32:11:56")
>>>> hdu.header.update('LNGOBS',"120:00:00")
>>>>
>>>> # (make any modifications to the header you might want to... e.g., 
>>>> see Megan Sosey's
>>>> # email of April 8 for examples)
>>>>
>>>> # ... and save the data to disk as a FITS file:
>>>> hdu.writeto("test.fits")
>>>> print "Finished. See test.fits"
>>>> ===============End code============
>>>>
>>>> I'm not sure what the 'byte' stuff is about. It may be that it's not 
>>>> needed. I think the original 'raw' file is byte oriented--rather than 
>>>> int or whatever. Anyway,  I don't seem to have succeeded in getting 
>>>> header info into the file. Sort of. If I use DS9, I can see the image 
>>>> and the fits header with what I coded into it. Here's what I see with 
>>>> ds9.
>>>>
>>>> *However, with my ccdsoft program, te image is black and the header 
>>>> is not the same. Whoops. Interesting. I changed the file to test.fit 
>>>> instead of test.fits, and the header info is there, but the image is 
>>>> still black. The dimension are the same, 640x480. I'm attaching 
>>>> test-ww.fits. I just renamed my test.fits. I think I'll check with 
>>>> The Bisque (ccdsoft) to see what format they produce and use for the 
>>>> raw image.
>>>> Question. When I execute the program a second time, the 
>>>> *hdu.writeto("test.fits") *will not write over the old file. Is there 
>>>> an option to force the write?
>>>>
>>>> Well, this is all good progress. **  *
>>>>
>>>> -- 
>>>>
>>>>          Wayne Watson (Watson Adventures, Prop., Nevada City, CA)
>>>>
>>>>            (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
>>>>             Obz Site:  39° 15' 7" N, 121° 2' 32" W, 2700 feet
>>>>          All the neutrons, and protons in the human body occupy
>>>>          a cube whose side is 5.52*10**-6 meters (tiny!). That
>>>>          adds up to a 150 pound person. It's not a surprise that
>>>>          we are mostly space. (Calculation by WTW)
>>>>       
>>> =============================================================
>>> Peter Erwin                   Max-Planck-Insitute for Extraterrestrial
>>> erwin at mpe.mpg.de              Physics, Giessenbachstrasse
>>> tel. +49 (0)89 30000 3695     85748 Garching, Germany
>>> fax  +49 (0)89 30000 3495     http://www.mpe.mpg.de/~erwin
>>>
>>>
>>>
>>>     
>>
>>   
>

-- 
           Wayne Watson (Watson Adventures, Prop., Nevada City, CA)

             (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
              Obz Site:  39° 15' 7" N, 121° 2' 32" W, 2700 feet  

           All the neutrons, and protons in the human body occupy
           a cube whose side is 5.52*10**-6 meters (tiny!). That
           adds up to a 150 pound person. It's not a surprise that
           we are mostly space. (Calculation by WTW)