[AstroPy] Moving Ahead with Raw Image Conversion
Peter Erwin
erwin at mpe.mpg.de
Tue Apr 14 11:58:23 EDT 2009
A bit more background, which might explain some of the confusion:
Originally, there was a package called Numeric Python, which was
sometimes
unofficially called "NumPy".
A few years ago, some people (mostly at STScI, I think) created a
modified
version which they called numarray; the original versions of pyfits
used this,
which is why the version 1.0 manual of pyfits mentions numarray.
More recently, the numarray and Numeric Python folks agreed to merge
projects
and create an updated package, now called numpy. All the up-to-date
versions
of array-using packages, including pyfits, now use numpy.
(I'm probably glossing over some fine details, but I think that gives
you the basic
picture.)
So, as Jim said, just use numpy. (Things are much less confusing now
than they
were a couple of years ago, when numpy was new and numarray was still
being
supported...)
(You can still find code on the web that was written back when Numeric
Python
was the only thing around; *most* -- but not all -- Numeric functions,
object, etc.,
are the same as the current numpy versions, so you can often update
such code
to use numpy without much trouble.)
cheers,
Peter
On Apr 14, 2009, at 5:35 PM, Jim Vickroy wrote:
> At the risk of stating the obvious, you should only be using numpy
> (not numarray). Numarray was a competing package that pyfits
> previously supported. Numarray is deprecated and the recent
> releases of pyfits have complete support for numpy.
>
> -- jv
>
> Wayne Watson wrote:
>>
>> Howdy, I think I see the ccdsoft problem. I looked at a fit (fits)
>> file
>> I produced with my CCD camera using it, and it produces a uint16
>> (BITPIX=16). I just made the change you suggested, and the resulting
>> file shows BITPIX=8, but CCDSoft liked it. Apparently, it objected to
>> the 32 bit image earlier, which probably gave rise to the black
>> image.
>> The former way produced a file of 1,204KB, and the current file
>> produced
>> sized with the simple replacement produced a file of 304KB. I used
>> dtype='uint8' instead of 'B'.
>>
>> Somewhere along the line I confused myself with the numarray use in
>> the
>> FITS manual, and apparently its look-alike notion ( as seen via
>> arange
>> and reshape) in numpy. Maybe they are equivalent in every respect.
>> I was
>> trying to figure out how to get Python to put together a 'matrix'
>> (mathematical sense) to feed it to data, but it looks like reshape is
>> the ticket. reshape seems missing from numarray (not mentioned in the
>> fits manual), so maybe there is a difference. I'm not sure the pyfits
>> manual ever showed how to make a 'matrix' and stuff it into the data
>> portion of a FITS header (maybe that's what section 2.2.3 is about,
>> which interestingly has the same title as 2.2.2).
>>
>> I think we are pretty much, successfully, done here. Allowing the
>> meteor
>> program I'm adding new features to be able to use FITS will be a very
>> useful change. Some exposure to pyfits and numpy helps too from the
>> coding aspects. Of course, if I decide to add features that allow
>> FITS
>> files into the program, say, for IP (img processing) work, that'll
>> require some unraveling of the data part of the fits file for
>> PIL/Tinter. Later though.
>>
>> Thanks for the help.
>>
>> Peter Erwin wrote:
>>
>>> Hi Wayne,
>>>
>>> Anne Archibald pointed out a much more concise and elegant way to
>>> read in
>>> and convert the raw file, using a function in Numpy. So you could
>>> replace the
>>> following original code:
>>>
>>>
>>>> # get raw data from .raw file:
>>>> raw_file=open('sent_internal.raw','rb')
>>>> raw_image=raw_file.read()
>>>> raw_file.close()
>>>>
>>>> # convert raw data to Python list of integers (1-D):
>>>> rawim_intarray = []
>>>> for x in raw_image:
>>>> # convert byte to integer ('B' = treat data as single unsigned
>>>> byte)
>>>> newint = struct.unpack('B', x)[0]
>>>> rawim_intarray.append(newint)
>>>>
>>>> # convert Python list of integers into numpy array of integers
>>>> (note:
>>>> still 1-D at this point)
>>>> rawim_numpy = numpy.array(rawim_intarray)
>>>>
>>> with this:
>>>
>>> rawim_numpy = numpy.fromfile('sent_internal.raw', dtype='B')
>>>
>>> [note: dtype='B' is identical to dtype=numpy.uint8, which Anne
>>> used in
>>> her example]
>>>
>>> and then continue as before.
>>>
>>> This has the added benefit of producing a smaller output file (300
>>> KB vs
>>> 1.2 MB), since the numpy array is now explicitly in 8-bit-integer
>>> form
>>> (rather than the 32-bit-integer form that my code created), and so
>>> Pyfits will automatically save it as 8-bit integers (you can check
>>> this
>>> by looking at the header keyword BITPIX).
>>>
>>>
>>> You're right, the raw image is a sequence of (standard) 8-bit bytes.
>>> Well, all
>>> computer files are streams of bytes; the question is how to
>>> interpret
>>> them. Since
>>> you said earlier that it was supposed to be 8-bit values, the idea
>>> is
>>> then to
>>> interpret each individual byte as an unsigned "integer" value
>>> (values
>>> from 0 to 255);
>>> that's what the "dtype='B'" (or "dtype=numpy.uint8") option passed
>>> to
>>> numpy.fromfile()
>>> does. (My previous code did the same thing, but then assigned each
>>> value to a
>>> standard Python integer, which is 4 bytes long; perfectly workable,
>>> but a bit
>>> wasteful!).
>>>
>>> Anyway, when I do this, I can see an image in DS9 (a circular
>>> aperture
>>> with ragged
>>> edges and an overall horizontal gradient inside the aperture). So
>>> I'm
>>> pretty sure
>>> it's working. (I've attached a jpeg saved from DS9 to show what the
>>> image looks like.)
>>>
>>> I don't think I can help you with this "ccdsoft" program; I'm not
>>> familiar with it.
>>> If it's a Windows program, then it may expect all FITS files to
>>> end in
>>> ".fit" instead
>>> of ".fits", just because Windows traditionally expects files to have
>>> 3-letter extensions.
>>>
>>>
>>> As for overwriting an existing FITS file, try this:
>>>
>>> hdu.writeto("existing_file_name.fit", clobber=True)
>>>
>>> cheers,
>>>
>>> Peter
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>>
>>>
>>> On Apr 14, 2009, at 6:28 AM, Wayne Watson wrote:
>>>
>>>
>>>> Hi, Peter. It looks like I confused a few people with the hex. I
>>>> had
>>>> removed it, but only after I placed the code in the msg. I was
>>>> playing with hex in the event I might need it somehow. int was fine
>>>> for my experimental purposes.
>>>>
>>>> Here's your code with a minor correction. I changed this line:
>>>> rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>>> The difference is rawmin_numpy.
>>>> I added a print at the end, and threw in two lines that attempt to
>>>> put header info into the final file. Continuing below the code.
>>>>
>>>> =====================Start===============
>>>> import struct, numpy, pyfits # struct is part of the main Python
>>>> library
>>>>
>>>> # get raw data from .raw file:
>>>> raw_file=open('sent_internal.raw','rb')
>>>> raw_image=raw_file.read()
>>>> raw_file.close()
>>>>
>>>> # convert raw data to Python list of integers (1-D):
>>>> rawim_intarray = []
>>>> for x in raw_image:
>>>> # convert byte to integer ('B' = treat data as single unsigned
>>>> byte)
>>>> newint = struct.unpack('B', x)[0]
>>>> rawim_intarray.append(newint)
>>>>
>>>> # convert Python list of integers into numpy array of integers
>>>> (note:
>>>> still 1-D at this point)
>>>> rawim_numpy = numpy.array(rawim_intarray)
>>>>
>>>> # reshape numpy array into 2D form, using our knowledge of the
>>>> original image's
>>>> # x and y sizes (kudos to Megan Sosey for pointing out how to do
>>>> this)
>>>> rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>>>
>>>> # create Pyfits header-data unit:
>>>> hdu = pyfits.PrimaryHDU(rawim_numpy_2d)
>>>> hdu.header.update('LATOBS',"32:11:56")
>>>> hdu.header.update('LNGOBS',"120:00:00")
>>>>
>>>> # (make any modifications to the header you might want to... e.g.,
>>>> see Megan Sosey's
>>>> # email of April 8 for examples)
>>>>
>>>> # ... and save the data to disk as a FITS file:
>>>> hdu.writeto("test.fits")
>>>> print "Finished. See test.fits"
>>>> ===============End code============
>>>>
>>>> I'm not sure what the 'byte' stuff is about. It may be that it's
>>>> not
>>>> needed. I think the original 'raw' file is byte oriented--rather
>>>> than
>>>> int or whatever. Anyway, I don't seem to have succeeded in getting
>>>> header info into the file. Sort of. If I use DS9, I can see the
>>>> image
>>>> and the fits header with what I coded into it. Here's what I see
>>>> with
>>>> ds9.
>>>>
>>>> *However, with my ccdsoft program, te image is black and the header
>>>> is not the same. Whoops. Interesting. I changed the file to
>>>> test.fit
>>>> instead of test.fits, and the header info is there, but the image
>>>> is
>>>> still black. The dimension are the same, 640x480. I'm attaching
>>>> test-ww.fits. I just renamed my test.fits. I think I'll check with
>>>> The Bisque (ccdsoft) to see what format they produce and use for
>>>> the
>>>> raw image.
>>>> Question. When I execute the program a second time, the
>>>> *hdu.writeto("test.fits") *will not write over the old file. Is
>>>> there
>>>> an option to force the write?
>>>>
>>>> Well, this is all good progress. ** *
>>>>
>>>> --
>>>>
>>>> Wayne Watson (Watson Adventures, Prop., Nevada City, CA)
>>>>
>>>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
>>>> Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet
>>>> All the neutrons, and protons in the human body occupy
>>>> a cube whose side is 5.52*10**-6 meters (tiny!). That
>>>> adds up to a 150 pound person. It's not a surprise that
>>>> we are mostly space. (Calculation by WTW)
>>>>
>>> =============================================================
>>> Peter Erwin Max-Planck-Insitute for
>>> Extraterrestrial
>>> erwin at mpe.mpg.de Physics, Giessenbachstrasse
>>> tel. +49 (0)89 30000 3695 85748 Garching, Germany
>>> fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin
>>>
>>>
>>>
>>>
>>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
=============================================================
Peter Erwin Max-Planck-Insitute for Extraterrestrial
erwin at mpe.mpg.de Physics, Giessenbachstrasse
tel. +49 (0)89 30000 3695 85748 Garching, Germany
fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin
More information about the AstroPy
mailing list