<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
FONT-SIZE: 10pt;
FONT-FAMILY:Tahoma
}
</style>
</head>
<body class='hmmessage'>Thanks. I'll churn on this for awhile..<BR><BR><BR><BR>
<HR id=stopSpelling>
> Date: Thu, 15 May 2008 16:00:03 -0700<BR>> From: gherron@islandtraining.com<BR>> To: python-list@python.org<BR>> Subject: Re: How do I use the unpack function?<BR>> <BR>> Marlin Rowley wrote:<BR>> > Hey Gary!<BR>> <BR>> Please keep such discussions on the public python-list -- not personal <BR>> e-mail.<BR>> <BR>> Scroll down for an answer to your latest question.<BR>> ><BR>> > Here's what I have that renders fine but I see some optimization that <BR>> > can be done (as you mentioned):<BR>> ><BR>> > # Tile Generation<BR>> > # This is where all the drawing to the client window<BR>> > # will happen. <BR>> > def generateTile( txl, tyl, tileWidth, tileHeight, clientWindow ):<BR>> > # make rgba (8-bit) data structure and zero it out.<BR>> > rgb = zeros( tileWidth*tileHeight*3, UnsignedInt8 )<BR>> > alpha = zeros( tileWidth*tileHeight*3, UnsignedInt8 )<BR>> > <BR>> > #print 'tileWidth: %s' % tileWidth<BR>> > #print 'tileHeight: %s' % tileHeight<BR>> > # for each pixel in the tile<BR>> > # we must invert the rendering of each<BR>> > # tile for wxPython's Bitmap support.<BR>> > for y in range( (tileHeight-1),-1,-1 ):<BR>> > for color in range(4):<BR>> > <BR>> > # read per scanline<BR>> > pixelComp = clientWindow.fileIO.read(4*tileWidth) <<<< <BR>> > HERE'S YOUR OPTIMIZATION!!<BR>> > <BR>> > for x in range(tileWidth):<BR>> > # mental ray streams RGBA components across the width<BR>> > # of every tile. so it first does all the r's,<BR>> > # then all the g's, then all the b's, etc.. across<BR>> > # the width.. Then it streams the second row, etc..<BR>> > # However, wxPython Bitmap class accepts an array of<BR>> > # tuples or just a byte order of RGBARGBARGBA, etc..<BR>> > # so we convert, keeping track of an offset.<BR>> > if color < 3:<BR>> > index = (3*(y*tileWidth+x))+color<BR>> > else:<BR>> > index = (3*(y*tileWidth+x))<BR>> > <BR>> > <BR>> > # RGBA_FP<BR>> > if clientWindow.pixelCode == 13:<BR>> > <BR>> > # unpack the pixel<BR>> > #fourbytes = pixelComp[:4]<BR>> > #pixelComp = pixelComp[4:]<BR>> > buffer = unpack("!f", pixelComp[4*x:4*x+4]) <BR>> > <<<<<<<<<<<<<<<<<< YOUR OPTIMIZATION!!<BR>> > <BR>> > # convert from 32-bit to 8-bit precision<BR>> > gamma = clientWindow.gamma<BR>> > if gamma == 1.0:<BR>> > pixel = int(255 * buffer[0] + 0.5)<BR>> > pixel = clamp(pixel,0,255)<BR>> > if color == 3:<BR>> > alpha[index+0] = alpha[index+1] = <BR>> > alpha[index+2] = pixel<BR>> > else:<BR>> > rgb[index] = pixel<BR>> > else:<BR>> > pixel = int(buffer[0] * GAMMA_BIT_PRECISION + 0.5)<BR>> > pixel = clamp(pixel,0,GAMMA_BIT_PRECISION-1)<BR>> > # set the color and alpha<BR>> > if color == 3:<BR>> > alpha[index+0] = alpha[index+1] = <BR>> > alpha[index+2] = clientWindow.frame.gammaTable[pixel]<BR>> > else:<BR>> > rgb[index] = <BR>> > clientWindow.frame.gammaTable[pixel]<BR>> > <BR>> > <BR>> > # ...<BR>> ><BR>> > # create an empty rgb and alpha tile<BR>> > tileRGB = wx.BitmapFromBuffer( tileWidth, tileHeight, rgb )<BR>> > tileAlpha = wx.BitmapFromBuffer( tileWidth, tileHeight, alpha )<BR>> > <BR>> > # set up main device to render to the current<BR>> > # buffers<BR>> > dc = wx.BufferedDC( None,clientWindow.colorBuffer )<BR>> > dca = wx.BufferedDC( None,clientWindow.alphaBuffer )<BR>> > <BR>> > # draw tiles<BR>> > dc.DrawBitmap( tileRGB, txl, (clientWindow.height-tileHeight)-tyl )<BR>> > dca.DrawBitmap( tileAlpha, txl, (clientWindow.height-tileHeight)-tyl )<BR>> ><BR>> ><BR>> > I'm no python expert (as you can tell), but I'm trying.. :)<BR>> ><BR>> > I started to re-write this function but I'm confused on how to do:<BR>> ><BR>> > > Reshape the array into a 3D array (i.e., a rectangular array of RGBA<BR>> > > values)<BR>> ><BR>> > this without using a for loop.<BR>> <BR>> Yes, easily. No Python loops (although plenty of C-level loops.) (I <BR>> think you are using Numeric -- a ancient predecessor of numpy. However, <BR>> I think these operations work in Numeric.)<BR>> <BR>> <BR>> Again I create a small test case. The values will be in the order <BR>> rrrrggggbbbb to start with, and rgbrgbrgbrgb afterwards.<BR>> <BR>> Create a test array and examine it:<BR>> >>> a = numpy.frombuffer('rrrrggggbbbb', dtype='S1')<BR>> >>> a<BR>> array(['r', 'r', 'r', 'r', 'g', 'g', 'g', 'g', 'b', 'b', 'b', 'b'], <BR>> dtype='|S1')<BR>> <BR>> Isolate each color component (here height*width = 4)<BR>> >>> a.shape = (3,4)<BR>> >>> a<BR>> array([['r', 'r', 'r', 'r'],<BR>> ['g', 'g', 'g', 'g'],<BR>> ['b', 'b', 'b', 'b']],<BR>> dtype='|S1')<BR>> <BR>> Transpose it. (This creates a new array by copying efficiently.)<BR>> >>> b = a.transpose()<BR>> >>> b<BR>> array([['r', 'g', 'b'],<BR>> ['r', 'g', 'b'],<BR>> ['r', 'g', 'b'],<BR>> ['r', 'g', 'b']],<BR>> dtype='|S1')<BR>> <BR>> Reset it's shape to be a width*height array of rgb's.<BR>> >>> b.shape = (2,2,3)<BR>> >>> b<BR>> array([[['r', 'g', 'b'],<BR>> ['r', 'g', 'b']],<BR>> <BR>> [['r', 'g', 'b'],<BR>> ['r', 'g', 'b']]],<BR>> dtype='|S1')<BR>> <BR>> Put it out in byte array form:<BR>> >>> b.tostring()<BR>> 'rgbrgbrgbrgb'<BR>> <BR>> <BR>> Done.<BR>> <BR>> <BR>> Gary Herron<BR>> <BR>> <BR>> ><BR>> > -M<BR>> ><BR>> ><BR>> > > Date: Thu, 15 May 2008 13:46:06 -0700<BR>> > > From: gherron@islandtraining.com<BR>> > > CC: python-list@python.org<BR>> > > Subject: Re: How do I use the unpack function?<BR>> > ><BR>> > > Marlin Rowley wrote:<BR>> > > > Gary,<BR>> > > ><BR>> > > > I'm getting streaming tile data from a renderer in order to allow the<BR>> > > > user to see the rendering of tiles in real-time to create the total<BR>> > > > image. I'm NOT reading an entire image scanline-by-scanline. The<BR>> > > > renderer streams in a series of floats (for each tile) and I build<BR>> > > > this tile up from each individual color component passed in. I then<BR>> > > > convert it to a small bitmap and blit() it to the window that will<BR>> > > > eventually make up my entire image. I'm just wanting to make the<BR>> > > > process as fast as the renderer can render the tiles. I've noticed on<BR>> > > > scenes that aren't very complex for the renderer that my python <BR>> > script<BR>> > > > is spending most of it's time drawing while the renderer is already<BR>> > > > done. What you've given me has sped up the drawing a lot, but I'm<BR>> > > > hungry for more optimization!<BR>> > > ><BR>> > > > There is no file format to be read. The data is raw bytes and can be<BR>> > > > any format.<BR>> > ><BR>> > > You are misinterpreting what I mean by a format. All data is a string<BR>> > > of bytes, and interpreting that string of bytes to have some particular<BR>> > > form is applying a format to it. Your particular format is: each 4<BR>> > > bytes represents a float, and a string of such floats give the RGBA<BR>> > > values for a rectangle (of some size) of pixels. I doubt that you are<BR>> > > the first ever to use that particular format to encode an image, but I<BR>> > > also don't believe it matches any of the standard image formats.<BR>> > ><BR>> > > So... There is still hope to use already written tools to decode your<BR>> > > format.<BR>> > ><BR>> > > Here's a hint on how to use numpy for decoding a byte string into an<BR>> > > array of floats. My example byte string is a hand coded string of just<BR>> > > 12 bytes -- you should replace that with a whole row or better yet, a<BR>> > > whole tile's worth of bytes.<BR>> > ><BR>> > > >>> import numpy<BR>> > > >>> byteString = '\x00\x00\x80?\x00\x00\x00@\x00\x00@@'<BR>> > > >>> b = numpy.frombuffer(byteString, dtype=numpy.float32)<BR>> > > >>> b<BR>> > > array([ 1., 2., 3.], dtype=float32)<BR>> > ><BR>> > ><BR>> > > If you want to use the array module instead:<BR>> > ><BR>> > > >>> import array<BR>> > > >>> a = array.array('f')<BR>> > > >>> a.fromstring(byteString)<BR>> > > >>> a<BR>> > > array('f', [1.0, 2.0, 3.0])<BR>> > ><BR>> > ><BR>> > > In either case ANY number of bytes can be decoded into an array of<BR>> > > floats, in one highly efficient call from Python.<BR>> > ><BR>> > > What you do with that array of float afterwards is up to you....<BR>> > ><BR>> > > If I read your note correctly, here's what I'd do to turn the array of<BR>> > > bytes into an image of a tile to be displayed on the screen.<BR>> > ><BR>> > > Get the whole tile's worth of bytes with one read into a single string.<BR>> > > Decode that string into an array of floats (as above).<BR>> > > Reshape the array into a 3D array (i.e., a rectangular array of RGBA<BR>> > > values)<BR>> > > Convert that array of floats into an array of 8-bit integers (scaling<BR>> > > all by 255).<BR>> > > Extract (via tostring) that array into a string of bytes.<BR>> > > Send that string of bytes to the graphics system as an RGBA array of<BR>> > > pixels.<BR>> > ><BR>> > > Each of these calls is one Python call into a highly efficient library.<BR>> > > This is using Python as a, so called, glue language.<BR>> > ><BR>> > > Gary Herron<BR>> > ><BR>> > ><BR>> > ><BR>> > ><BR>> > ><BR>> > ><BR>> > ><BR>> > > ><BR>> > > > > Date: Thu, 15 May 2008 10:09:45 -0700<BR>> > > > > From: gherron@islandtraining.com<BR>> > > > > CC: python-list@python.org<BR>> > > > > Subject: Re: How do I use the unpack function?<BR>> > > > ><BR>> > > > > John Machin wrote:<BR>> > > > > > On May 16, 2:11 am, Gary Herron <gher...@islandtraining.com> <BR>> > wrote:<BR>> > > > > ><BR>> > > > > >> Marlin Rowley wrote:<BR>> > > > > >><BR>> > > > > >>> All:<BR>> > > > > >>><BR>> > > > > >>> I've got a script that runs really slow because I'm reading <BR>> > from a<BR>> > > > > >>> stream a byte at a time:<BR>> > > > > >>><BR>> > > > > >>> // TERRIBLE<BR>> > > > > >>> for y in range( height ):<BR>> > > > > >>> for color in range(4):<BR>> > > > > >>> for x in range( width ):<BR>> > > > > >>> pixelComponent = fileIO.read(4)<BR>> > > > > >>> buffer = unpack("!f",pixelComponent) << unpacks ONE<BR>> > > > > >>><BR>> > > > > ><BR>> > > > > > [snip]<BR>> > > > > > Perhaps the OP might be able to use the Python Imaging Library <BR>> > (PIL)<BR>> > > > > > instead of reinventing an image-file handler.<BR>> > > > > ><BR>> > > > ><BR>> > > > > Indeed. That's why my original answer included the line:<BR>> > > > > There are probably better ways overall, but this directly answers<BR>> > > > > your question.<BR>> > > > ><BR>> > > > > Other possibilities.<BR>> > > > ><BR>> > > > > I don't recognize the file format being read in here, but if it is a<BR>> > > > > standard image format, the PIL suggestion is a good way to go.<BR>> > > > ><BR>> > > > > Or, if it is an image of not too enormous size, read the *whole* <BR>> > thing<BR>> > > > > in at once.<BR>> > > > ><BR>> > > > > Or rather than manipulate the array by carving off 4 bytes at a <BR>> > time,<BR>> > > > > just index through it in 4 byte chunks:<BR>> > > > > for i in range(width):<BR>> > > > > buffer = unpack("!f", pixelComponent[4*i:4*i+4])<BR>> > > > ><BR>> > > > > Or<BR>> > > > > for i in range(0,4*width,4):<BR>> > > > > buffer = unpack("!f", pixelComponent[i:i+4])<BR>> > > > ><BR>> > > > > Or<BR>> > > > > Use numpy. Create an array of floats, and initialize it with the <BR>> > byte<BR>> > > > > string, making sure to take endianess int account. (I'm quite <BR>> > sure this<BR>> > > > > could be made to work, and then the whole operation is <BR>> > enormously fast<BR>> > > > > with only several lines of Python code and *no* Python loops.<BR>> > > > ><BR>> > > > > Or<BR>> > > > > ...?<BR>> > > > ><BR>> > > > > Gary Herron<BR>> > > > ><BR>> > > > > > --<BR>> > > > > > http://mail.python.org/mailman/listinfo/python-list<BR>> > > > > ><BR>> > > > ><BR>> > > > > --<BR>> > > > > http://mail.python.org/mailman/listinfo/python-list<BR>> > > ><BR>> > > > <BR>> > ------------------------------------------------------------------------<BR>> > > > Windows Live SkyDrive lets you share files with faraway friends. <BR>> > Start<BR>> > > > sharing.<BR>> > > > <BR>> > <http://www.windowslive.com/skydrive/overview.html?ocid=TXT_TAGLM_WL_Refresh_skydrive_052008> <BR>> ><BR>> > > ><BR>> > > > <BR>> > ------------------------------------------------------------------------<BR>> > > ><BR>> > > > --<BR>> > > > http://mail.python.org/mailman/listinfo/python-list<BR>> > ><BR>> > > --<BR>> > > http://mail.python.org/mailman/listinfo/python-list<BR>> ><BR>> > ------------------------------------------------------------------------<BR>> > Get Free (PRODUCT) RED™ Emoticons, Winks and Display Pics. Check it <BR>> > out! <BR>> > <http://joinred.spaces.live.com?ocid=TXT_HMTG_prodredemoticons_052008><BR>> <BR>> --<BR>> http://mail.python.org/mailman/listinfo/python-list<BR><br /><hr />Stay in touch when you're away with Windows Live Messenger. <a href='http://www.windowslive.com/messenger/overview.html?ocid=TXT_TAGLM_WL_Refresh_messenger_052008' target='_new'>IM anytime you're online.</a></body>
</html>