Help mi please with optimalization of simple Python code

[Jeff Epler]

On Wed, Oct 02, 2002 at 02:25:47PM +0200, Dalibor Jelinek wrote:
> Hi Jeff,
> thank you for your will to help me.
> I rewrote my script using your suggestion
> but to my big suprprise it was much worse!
> My orginal script took - 28 sec
> Scipt with array.fromstring - 499 sec
> Scipt with array.fromfile - 480 sec
> 
> Do you have some idea what's wrong?
> Thanks for answer in advace.

Wow!  That's impressively slower!

Have you tried the suggestion of using struct.unpack to get your values?

If I had to guess, I'd say that for the expression 'l[x]' a list is
somewhat faster than an array.  For starters, there's a special case
(optimization) directly in the ceval loop for subscripts where l passes
'PyList_CheckExact' and x passes 'PyInt_CheckExact', while for an array
object there are more intermediate calls.  This will hurt you more and
more if you repeatedly reference 'l[x]' instead of doing 'val = l[x]'
and then repeatedly referencing 'val'.

However, the results I've gotten (where "... process val" is just
'pass', and the file read is /dev/zero, a file full of '\0' bytes) don't
agree with yours (and "array"'s lead over "struct" and "ord" seems to
only increase as the dataset gets larger).  Timings in seconds for NxN
grids of varying sizes:
	   struct    array      ord
     256    0.156    0.083    0.413
     512    0.589    0.295    1.652
    1024    2.304    1.119    6.664
    4096   37.121   17.143  135.111


The program:

import array, struct


def x1(f, d):
    range_maxX = range_maxY = range(d)
    format = "<%dH" % d
    sz = struct.calcsize(format)
    for y in range_maxY:
	line = struct.unpack(format, f.read(sz))
	for val in line: pass

def x2(f, d):
    range_maxX = range_maxY = range(d)
    for y in range_maxY:
	line = array.array('H')
	line.fromstring(f.read(d*2))
	#line.byteswap()
	for val in line: pass

def x3(f, d):
    range_maxX = range_maxY = range(d)
    for y in range_maxY:
	line = f.read(d*2)
	for x in range_maxX:
	    val = ord(line[x]) + 256 * ord(line[x+1])
	    pass

import time
f = open("/dev/zero")
print "%4s %8s %8s %8s" % ("", "struct", "array", "ord")
for d in (256, 512, 1024, 4096):
    print "%4d" % d,
    for fun in x1, x2, x3:
	t0 = time.time()
	fun(f, d)
	t1 = time.time()
	print "%8.3f" % (t1-t0),
    print