[AstroPy] Moving Ahead with Raw Image Conversion

[Wayne Watson]

Howdy, I think I see the ccdsoft problem. I looked at a fit (fits) file 
I produced with my CCD camera using it, and it produces a uint16 
(BITPIX=16). I just made the change you suggested, and the resulting 
file shows BITPIX=8, but CCDSoft liked it. Apparently, it objected to 
the 32 bit image earlier, which probably gave rise to the black image. 
The former way produced a file of 1,204KB, and the current file produced 
sized with the simple replacement produced a file of 304KB. I used 
dtype='uint8' instead of 'B'.

Somewhere along the line I confused myself with the numarray use in the 
FITS manual, and apparently its look-alike notion ( as seen via arange 
and reshape) in numpy. Maybe they are equivalent in every respect. I was 
trying to figure out how to get Python to put together a 'matrix' 
(mathematical sense) to feed it to data, but it looks like reshape is 
the ticket. reshape seems missing from numarray (not mentioned in the 
fits manual), so maybe there is a difference. I'm not sure the pyfits 
manual ever showed how to make a 'matrix' and stuff it into the data 
portion of a FITS header (maybe that's what section 2.2.3 is about, 
which interestingly has the same title as 2.2.2).

I think we are pretty much, successfully, done here. Allowing the meteor 
program I'm adding new features to be able to use FITS will be a very 
useful change. Some exposure to pyfits and numpy helps too from the 
coding aspects. Of course, if I decide to add features that allow FITS 
files into the program, say, for IP (img processing) work, that'll 
require some unraveling of the data part of the fits file for 
PIL/Tinter. Later though.

Thanks for the help.

Peter Erwin wrote:
> Hi Wayne,
>
> Anne Archibald pointed out a much more concise and elegant way to read in
> and convert the raw file, using a function in Numpy.  So you could 
> replace the
> following original code:
>
>> # get raw data from .raw file:
>> raw_file=open('sent_internal.raw','rb')
>> raw_image=raw_file.read()
>> raw_file.close()
>>
>> # convert raw data to Python list of integers (1-D):
>> rawim_intarray = []
>> for x in raw_image:
>>   # convert byte to integer ('B' = treat data as single unsigned byte)
>>   newint = struct.unpack('B', x)[0]
>>   rawim_intarray.append(newint)
>>
>> # convert Python list of integers into numpy array of integers (note: 
>> still 1-D at this point)
>> rawim_numpy = numpy.array(rawim_intarray)
>
> with this:
>
> rawim_numpy = numpy.fromfile('sent_internal.raw', dtype='B')
>
> [note: dtype='B' is identical to dtype=numpy.uint8, which Anne used in 
> her example]
>
> and then continue as before.
>
> This has the added benefit of producing a smaller output file (300 KB vs
> 1.2 MB), since the numpy array is now explicitly in 8-bit-integer form
> (rather than the 32-bit-integer form that my code created), and so
> Pyfits will automatically save it as 8-bit integers (you can check this
> by looking at the header keyword BITPIX).
>
>
> You're right, the raw image is a sequence of (standard) 8-bit bytes.  
> Well, all
> computer files are streams of bytes; the question is how to interpret 
> them.  Since
> you said earlier that it was supposed to be 8-bit values, the idea is 
> then to
> interpret each individual byte as an unsigned "integer" value (values 
> from 0 to 255);
> that's what the "dtype='B'" (or "dtype=numpy.uint8") option passed to 
> numpy.fromfile()
> does.  (My previous code did the same thing, but then assigned each 
> value to a
> standard Python integer, which is 4 bytes long; perfectly workable, 
> but a bit
> wasteful!).
>
> Anyway, when I do this, I can see an image in DS9 (a circular aperture 
> with ragged
> edges and an overall horizontal gradient inside the aperture).  So I'm 
> pretty sure
> it's working.  (I've attached a jpeg saved from DS9 to show what the 
> image looks like.)
>
> I don't think I can help you with this "ccdsoft" program; I'm not 
> familiar with it.
> If it's a Windows program, then it may expect all FITS files to end in 
> ".fit" instead
> of ".fits", just because Windows traditionally expects files to have 
> 3-letter extensions.
>
>
> As for overwriting an existing FITS file, try this:
>
> hdu.writeto("existing_file_name.fit", clobber=True)
>
> cheers,
>
> Peter
>
>
>
> ------------------------------------------------------------------------
>
>
>
>
> On Apr 14, 2009, at 6:28 AM, Wayne Watson wrote:
>
>> Hi, Peter. It looks like I confused a few people with the hex. I had 
>> removed it, but only after I placed the code in the msg. I was 
>> playing with hex in the event I might need it somehow. int was fine 
>> for my experimental purposes.
>>
>> Here's your code with a minor correction. I changed this line:
>>  rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>> The difference is rawmin_numpy.
>> I added a print at the end, and threw in two lines that attempt to 
>> put header info into the final file.  Continuing below the code.
>>
>> =====================Start===============
>> import struct, numpy, pyfits   # struct is part of the main Python 
>> library
>>
>> # get raw data from .raw file:
>> raw_file=open('sent_internal.raw','rb')
>> raw_image=raw_file.read()
>> raw_file.close()
>>
>> # convert raw data to Python list of integers (1-D):
>> rawim_intarray = []
>> for x in raw_image:
>>   # convert byte to integer ('B' = treat data as single unsigned byte)
>>   newint = struct.unpack('B', x)[0]
>>   rawim_intarray.append(newint)
>>
>> # convert Python list of integers into numpy array of integers (note: 
>> still 1-D at this point)
>> rawim_numpy = numpy.array(rawim_intarray)
>>
>> # reshape numpy array into 2D form, using our knowledge of the 
>> original image's
>> # x and y sizes (kudos to Megan Sosey for pointing out how to do this)
>> rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>
>> # create Pyfits header-data unit:
>> hdu = pyfits.PrimaryHDU(rawim_numpy_2d)
>> hdu.header.update('LATOBS',"32:11:56")
>> hdu.header.update('LNGOBS',"120:00:00")
>>
>> # (make any modifications to the header you might want to... e.g., 
>> see Megan Sosey's
>> # email of April 8 for examples)
>>
>> # ... and save the data to disk as a FITS file:
>> hdu.writeto("test.fits")
>> print "Finished. See test.fits"
>> ===============End code============
>>
>> I'm not sure what the 'byte' stuff is about. It may be that it's not 
>> needed. I think the original 'raw' file is byte oriented--rather than 
>> int or whatever. Anyway,  I don't seem to have succeeded in getting 
>> header info into the file. Sort of. If I use DS9, I can see the image 
>> and the fits header with what I coded into it. Here's what I see with 
>> ds9.
>>
>> *However, with my ccdsoft program, te image is black and the header 
>> is not the same. Whoops. Interesting. I changed the file to test.fit 
>> instead of test.fits, and the header info is there, but the image is 
>> still black. The dimension are the same, 640x480. I'm attaching 
>> test-ww.fits. I just renamed my test.fits. I think I'll check with 
>> The Bisque (ccdsoft) to see what format they produce and use for the 
>> raw image.
>> Question. When I execute the program a second time, the 
>> *hdu.writeto("test.fits") *will not write over the old file. Is there 
>> an option to force the write?
>>
>> Well, this is all good progress. **  *
>>
>> -- 
>>
>>          Wayne Watson (Watson Adventures, Prop., Nevada City, CA)
>>
>>            (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
>>             Obz Site:  39° 15' 7" N, 121° 2' 32" W, 2700 feet
>>          All the neutrons, and protons in the human body occupy
>>          a cube whose side is 5.52*10**-6 meters (tiny!). That
>>          adds up to a 150 pound person. It's not a surprise that
>>          we are mostly space. (Calculation by WTW)
>
> =============================================================
> Peter Erwin                   Max-Planck-Insitute for Extraterrestrial
> erwin at mpe.mpg.de              Physics, Giessenbachstrasse
> tel. +49 (0)89 30000 3695     85748 Garching, Germany
> fax  +49 (0)89 30000 3495     http://www.mpe.mpg.de/~erwin
>
>
>

-- 
           Wayne Watson (Watson Adventures, Prop., Nevada City, CA)

             (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
              Obz Site:  39° 15' 7" N, 121° 2' 32" W, 2700 feet  

           All the neutrons, and protons in the human body occupy
           a cube whose side is 5.52*10**-6 meters (tiny!). That
           adds up to a 150 pound person. It's not a surprise that
           we are mostly space. (Calculation by WTW)