# [AstroPy] Moving Ahead with Raw Image Conversion

Peter Erwin erwin at mpe.mpg.de
Tue Apr 14 07:26:00 EDT 2009

```Hi Wayne,

Anne Archibald pointed out a much more concise and elegant way to read
in
and convert the raw file, using a function in Numpy.  So you could
replace the
following original code:

> # get raw data from .raw file:
> raw_file=open('sent_internal.raw','rb')
> raw_file.close()
>
> # convert raw data to Python list of integers (1-D):
> rawim_intarray = []
> for x in raw_image:
>   # convert byte to integer ('B' = treat data as single unsigned byte)
>   newint = struct.unpack('B', x)[0]
>   rawim_intarray.append(newint)
>
> # convert Python list of integers into numpy array of integers
> (note: still 1-D at this point)
> rawim_numpy = numpy.array(rawim_intarray)

with this:

rawim_numpy = numpy.fromfile('sent_internal.raw', dtype='B')

[note: dtype='B' is identical to dtype=numpy.uint8, which Anne used in
her example]

and then continue as before.

This has the added benefit of producing a smaller output file (300 KB vs
1.2 MB), since the numpy array is now explicitly in 8-bit-integer form
(rather than the 32-bit-integer form that my code created), and so
Pyfits will automatically save it as 8-bit integers (you can check this
by looking at the header keyword BITPIX).

You're right, the raw image is a sequence of (standard) 8-bit bytes.
Well, all
computer files are streams of bytes; the question is how to interpret
them.  Since
you said earlier that it was supposed to be 8-bit values, the idea is
then to
interpret each individual byte as an unsigned "integer" value (values
from 0 to 255);
that's what the "dtype='B'" (or "dtype=numpy.uint8") option passed to
numpy.fromfile()
does.  (My previous code did the same thing, but then assigned each
value to a
standard Python integer, which is 4 bytes long; perfectly workable,
but a bit
wasteful!).

Anyway, when I do this, I can see an image in DS9 (a circular aperture
with ragged
edges and an overall horizontal gradient inside the aperture).  So I'm
pretty sure
it's working.  (I've attached a jpeg saved from DS9 to show what the
image looks like.)

I don't think I can help you with this "ccdsoft" program; I'm not
familiar with it.
If it's a Windows program, then it may expect all FITS files to end in
of ".fits", just because Windows traditionally expects files to have 3-
letter extensions.

As for overwriting an existing FITS file, try this:

hdu.writeto("existing_file_name.fit", clobber=True)

cheers,

Peter

-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.jpg
Type: image/jpeg
Size: 19595 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/astropy/attachments/20090414/7464fb52/attachment.jpg>
-------------- next part --------------

On Apr 14, 2009, at 6:28 AM, Wayne Watson wrote:

> Hi, Peter. It looks like I confused a few people with the hex. I had
> removed it, but only after I placed the code in the msg. I was
> playing with hex in the event I might need it somehow. int was fine
> for my experimental purposes.
>
> Here's your code with a minor correction. I changed this line:
>  rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
> The difference is rawmin_numpy.
> I added a print at the end, and threw in two lines that attempt to
> put header info into the final file.  Continuing below the code.
>
> =====================Start===============
> import struct, numpy, pyfits   # struct is part of the main Python
> library
>
> # get raw data from .raw file:
> raw_file=open('sent_internal.raw','rb')
> raw_file.close()
>
> # convert raw data to Python list of integers (1-D):
> rawim_intarray = []
> for x in raw_image:
>   # convert byte to integer ('B' = treat data as single unsigned byte)
>   newint = struct.unpack('B', x)[0]
>   rawim_intarray.append(newint)
>
> # convert Python list of integers into numpy array of integers
> (note: still 1-D at this point)
> rawim_numpy = numpy.array(rawim_intarray)
>
> # reshape numpy array into 2D form, using our knowledge of the
> original image's
> # x and y sizes (kudos to Megan Sosey for pointing out how to do this)
> rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>
> # create Pyfits header-data unit:
> hdu = pyfits.PrimaryHDU(rawim_numpy_2d)
>
> # (make any modifications to the header you might want to... e.g.,
> see Megan Sosey's
> # email of April 8 for examples)
>
> # ... and save the data to disk as a FITS file:
> hdu.writeto("test.fits")
> print "Finished. See test.fits"
> ===============End code============
>
> I'm not sure what the 'byte' stuff is about. It may be that it's not
> needed. I think the original 'raw' file is byte oriented--rather
> than int or whatever. Anyway,  I don't seem to have succeeded in
> getting header info into the file. Sort of. If I use DS9, I can see
> the image and the fits header with what I coded into it. Here's what
> I see with ds9.
>
> *However, with my ccdsoft program, te image is black and the header
> is not the same. Whoops. Interesting. I changed the file to test.fit
> instead of test.fits, and the header info is there, but the image is
> still black. The dimension are the same, 640x480. I'm attaching test-
> ww.fits. I just renamed my test.fits. I think I'll check with The
> Bisque (ccdsoft) to see what format they produce and use for the raw
> image.
> Question. When I execute the program a second time, the
> *hdu.writeto("test.fits") *will not write over the old file. Is
> there an option to force the write?
>
> Well, this is all good progress. **  *
>
> --
>
>
>            (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
>             Obz Site:  39? 15' 7" N, 121? 2' 32" W, 2700 feet
>          All the neutrons, and protons in the human body occupy
>          a cube whose side is 5.52*10**-6 meters (tiny!). That
>          adds up to a 150 pound person. It's not a surprise that
>          we are mostly space. (Calculation by WTW)

=============================================================
Peter Erwin                   Max-Planck-Insitute for Extraterrestrial
erwin at mpe.mpg.de              Physics, Giessenbachstrasse
tel. +49 (0)89 30000 3695     85748 Garching, Germany
fax  +49 (0)89 30000 3495     http://www.mpe.mpg.de/~erwin

```