How do I use the unpack function?
Gary Herron
gherron at islandtraining.com
Thu May 15 19:00:03 EDT 2008
Marlin Rowley wrote:
> Hey Gary!
Please keep such discussions on the public python-list -- not personal
e-mail.
Scroll down for an answer to your latest question.
>
> Here's what I have that renders fine but I see some optimization that
> can be done (as you mentioned):
>
> # Tile Generation
> # This is where all the drawing to the client window
> # will happen.
> def generateTile( txl, tyl, tileWidth, tileHeight, clientWindow ):
> # make rgba (8-bit) data structure and zero it out.
> rgb = zeros( tileWidth*tileHeight*3, UnsignedInt8 )
> alpha = zeros( tileWidth*tileHeight*3, UnsignedInt8 )
>
> #print 'tileWidth: %s' % tileWidth
> #print 'tileHeight: %s' % tileHeight
> # for each pixel in the tile
> # we must invert the rendering of each
> # tile for wxPython's Bitmap support.
> for y in range( (tileHeight-1),-1,-1 ):
> for color in range(4):
>
> # read per scanline
> pixelComp = clientWindow.fileIO.read(4*tileWidth) <<<<
> HERE'S YOUR OPTIMIZATION!!
>
> for x in range(tileWidth):
> # mental ray streams RGBA components across the width
> # of every tile. so it first does all the r's,
> # then all the g's, then all the b's, etc.. across
> # the width.. Then it streams the second row, etc..
> # However, wxPython Bitmap class accepts an array of
> # tuples or just a byte order of RGBARGBARGBA, etc..
> # so we convert, keeping track of an offset.
> if color < 3:
> index = (3*(y*tileWidth+x))+color
> else:
> index = (3*(y*tileWidth+x))
>
>
> # RGBA_FP
> if clientWindow.pixelCode == 13:
>
> # unpack the pixel
> #fourbytes = pixelComp[:4]
> #pixelComp = pixelComp[4:]
> buffer = unpack("!f", pixelComp[4*x:4*x+4])
> <<<<<<<<<<<<<<<<<< YOUR OPTIMIZATION!!
>
> # convert from 32-bit to 8-bit precision
> gamma = clientWindow.gamma
> if gamma == 1.0:
> pixel = int(255 * buffer[0] + 0.5)
> pixel = clamp(pixel,0,255)
> if color == 3:
> alpha[index+0] = alpha[index+1] =
> alpha[index+2] = pixel
> else:
> rgb[index] = pixel
> else:
> pixel = int(buffer[0] * GAMMA_BIT_PRECISION + 0.5)
> pixel = clamp(pixel,0,GAMMA_BIT_PRECISION-1)
> # set the color and alpha
> if color == 3:
> alpha[index+0] = alpha[index+1] =
> alpha[index+2] = clientWindow.frame.gammaTable[pixel]
> else:
> rgb[index] =
> clientWindow.frame.gammaTable[pixel]
>
>
> # ...
>
> # create an empty rgb and alpha tile
> tileRGB = wx.BitmapFromBuffer( tileWidth, tileHeight, rgb )
> tileAlpha = wx.BitmapFromBuffer( tileWidth, tileHeight, alpha )
>
> # set up main device to render to the current
> # buffers
> dc = wx.BufferedDC( None,clientWindow.colorBuffer )
> dca = wx.BufferedDC( None,clientWindow.alphaBuffer )
>
> # draw tiles
> dc.DrawBitmap( tileRGB, txl, (clientWindow.height-tileHeight)-tyl )
> dca.DrawBitmap( tileAlpha, txl, (clientWindow.height-tileHeight)-tyl )
>
>
> I'm no python expert (as you can tell), but I'm trying.. :)
>
> I started to re-write this function but I'm confused on how to do:
>
> > Reshape the array into a 3D array (i.e., a rectangular array of RGBA
> > values)
>
> this without using a for loop.
Yes, easily. No Python loops (although plenty of C-level loops.) (I
think you are using Numeric -- a ancient predecessor of numpy. However,
I think these operations work in Numeric.)
Again I create a small test case. The values will be in the order
rrrrggggbbbb to start with, and rgbrgbrgbrgb afterwards.
Create a test array and examine it:
>>> a = numpy.frombuffer('rrrrggggbbbb', dtype='S1')
>>> a
array(['r', 'r', 'r', 'r', 'g', 'g', 'g', 'g', 'b', 'b', 'b', 'b'],
dtype='|S1')
Isolate each color component (here height*width = 4)
>>> a.shape = (3,4)
>>> a
array([['r', 'r', 'r', 'r'],
['g', 'g', 'g', 'g'],
['b', 'b', 'b', 'b']],
dtype='|S1')
Transpose it. (This creates a new array by copying efficiently.)
>>> b = a.transpose()
>>> b
array([['r', 'g', 'b'],
['r', 'g', 'b'],
['r', 'g', 'b'],
['r', 'g', 'b']],
dtype='|S1')
Reset it's shape to be a width*height array of rgb's.
>>> b.shape = (2,2,3)
>>> b
array([[['r', 'g', 'b'],
['r', 'g', 'b']],
[['r', 'g', 'b'],
['r', 'g', 'b']]],
dtype='|S1')
Put it out in byte array form:
>>> b.tostring()
'rgbrgbrgbrgb'
Done.
Gary Herron
>
> -M
>
>
> > Date: Thu, 15 May 2008 13:46:06 -0700
> > From: gherron at islandtraining.com
> > CC: python-list at python.org
> > Subject: Re: How do I use the unpack function?
> >
> > Marlin Rowley wrote:
> > > Gary,
> > >
> > > I'm getting streaming tile data from a renderer in order to allow the
> > > user to see the rendering of tiles in real-time to create the total
> > > image. I'm NOT reading an entire image scanline-by-scanline. The
> > > renderer streams in a series of floats (for each tile) and I build
> > > this tile up from each individual color component passed in. I then
> > > convert it to a small bitmap and blit() it to the window that will
> > > eventually make up my entire image. I'm just wanting to make the
> > > process as fast as the renderer can render the tiles. I've noticed on
> > > scenes that aren't very complex for the renderer that my python
> script
> > > is spending most of it's time drawing while the renderer is already
> > > done. What you've given me has sped up the drawing a lot, but I'm
> > > hungry for more optimization!
> > >
> > > There is no file format to be read. The data is raw bytes and can be
> > > any format.
> >
> > You are misinterpreting what I mean by a format. All data is a string
> > of bytes, and interpreting that string of bytes to have some particular
> > form is applying a format to it. Your particular format is: each 4
> > bytes represents a float, and a string of such floats give the RGBA
> > values for a rectangle (of some size) of pixels. I doubt that you are
> > the first ever to use that particular format to encode an image, but I
> > also don't believe it matches any of the standard image formats.
> >
> > So... There is still hope to use already written tools to decode your
> > format.
> >
> > Here's a hint on how to use numpy for decoding a byte string into an
> > array of floats. My example byte string is a hand coded string of just
> > 12 bytes -- you should replace that with a whole row or better yet, a
> > whole tile's worth of bytes.
> >
> > >>> import numpy
> > >>> byteString = '\x00\x00\x80?\x00\x00\x00@\x00\x00@@'
> > >>> b = numpy.frombuffer(byteString, dtype=numpy.float32)
> > >>> b
> > array([ 1., 2., 3.], dtype=float32)
> >
> >
> > If you want to use the array module instead:
> >
> > >>> import array
> > >>> a = array.array('f')
> > >>> a.fromstring(byteString)
> > >>> a
> > array('f', [1.0, 2.0, 3.0])
> >
> >
> > In either case ANY number of bytes can be decoded into an array of
> > floats, in one highly efficient call from Python.
> >
> > What you do with that array of float afterwards is up to you....
> >
> > If I read your note correctly, here's what I'd do to turn the array of
> > bytes into an image of a tile to be displayed on the screen.
> >
> > Get the whole tile's worth of bytes with one read into a single string.
> > Decode that string into an array of floats (as above).
> > Reshape the array into a 3D array (i.e., a rectangular array of RGBA
> > values)
> > Convert that array of floats into an array of 8-bit integers (scaling
> > all by 255).
> > Extract (via tostring) that array into a string of bytes.
> > Send that string of bytes to the graphics system as an RGBA array of
> > pixels.
> >
> > Each of these calls is one Python call into a highly efficient library.
> > This is using Python as a, so called, glue language.
> >
> > Gary Herron
> >
> >
> >
> >
> >
> >
> >
> > >
> > > > Date: Thu, 15 May 2008 10:09:45 -0700
> > > > From: gherron at islandtraining.com
> > > > CC: python-list at python.org
> > > > Subject: Re: How do I use the unpack function?
> > > >
> > > > John Machin wrote:
> > > > > On May 16, 2:11 am, Gary Herron <gher... at islandtraining.com>
> wrote:
> > > > >
> > > > >> Marlin Rowley wrote:
> > > > >>
> > > > >>> All:
> > > > >>>
> > > > >>> I've got a script that runs really slow because I'm reading
> from a
> > > > >>> stream a byte at a time:
> > > > >>>
> > > > >>> // TERRIBLE
> > > > >>> for y in range( height ):
> > > > >>> for color in range(4):
> > > > >>> for x in range( width ):
> > > > >>> pixelComponent = fileIO.read(4)
> > > > >>> buffer = unpack("!f",pixelComponent) << unpacks ONE
> > > > >>>
> > > > >
> > > > > [snip]
> > > > > Perhaps the OP might be able to use the Python Imaging Library
> (PIL)
> > > > > instead of reinventing an image-file handler.
> > > > >
> > > >
> > > > Indeed. That's why my original answer included the line:
> > > > There are probably better ways overall, but this directly answers
> > > > your question.
> > > >
> > > > Other possibilities.
> > > >
> > > > I don't recognize the file format being read in here, but if it is a
> > > > standard image format, the PIL suggestion is a good way to go.
> > > >
> > > > Or, if it is an image of not too enormous size, read the *whole*
> thing
> > > > in at once.
> > > >
> > > > Or rather than manipulate the array by carving off 4 bytes at a
> time,
> > > > just index through it in 4 byte chunks:
> > > > for i in range(width):
> > > > buffer = unpack("!f", pixelComponent[4*i:4*i+4])
> > > >
> > > > Or
> > > > for i in range(0,4*width,4):
> > > > buffer = unpack("!f", pixelComponent[i:i+4])
> > > >
> > > > Or
> > > > Use numpy. Create an array of floats, and initialize it with the
> byte
> > > > string, making sure to take endianess int account. (I'm quite
> sure this
> > > > could be made to work, and then the whole operation is
> enormously fast
> > > > with only several lines of Python code and *no* Python loops.
> > > >
> > > > Or
> > > > ...?
> > > >
> > > > Gary Herron
> > > >
> > > > > --
> > > > > http://mail.python.org/mailman/listinfo/python-list
> > > > >
> > > >
> > > > --
> > > > http://mail.python.org/mailman/listinfo/python-list
> > >
> > >
> ------------------------------------------------------------------------
> > > Windows Live SkyDrive lets you share files with faraway friends.
> Start
> > > sharing.
> > >
> <http://www.windowslive.com/skydrive/overview.html?ocid=TXT_TAGLM_WL_Refresh_skydrive_052008>
>
> > >
> > >
> ------------------------------------------------------------------------
> > >
> > > --
> > > http://mail.python.org/mailman/listinfo/python-list
> >
> > --
> > http://mail.python.org/mailman/listinfo/python-list
>
> ------------------------------------------------------------------------
> Get Free (PRODUCT) RED™ Emoticons, Winks and Display Pics. Check it
> out!
> <http://joinred.spaces.live.com?ocid=TXT_HMTG_prodredemoticons_052008>
More information about the Python-list
mailing list