summing NumPy byte arrays

[Chris Barker]

Fernando Pérez wrote:
> Just curious: why not call reduce() with an object whose __add__
> method does the work of adding an int +byte as int? The loop would be
> done in C (by reduce) and the tying should work ok.
> 
> Maybe I'm missing something,

We use Numeric arrays because they provide very nice notation, and
because they are MUCH faster for crunching a lot of number. I assume
that if the OP is conserned about memory enough that he wants to use
1byte integers, he has a LOT of numbers to work with. An example:
#!/usr/bin/env python

from Numeric import *
import  RandomArray 
import time

# create a long 1byte integer array
n = 500000
m = 10
X = RandomArray.randint(0,255,(n,)).astype('b')

start = time.clock()
for i in range(m): # do it in loop to make it take longer
    s = sum(X)
print "it look %f seconds to use Numeric.sum"%(time.clock()-start)
print s
# this is fast, but it gets the wrong answer becasuse the answer won't
fit into 1-byte.

# and with reduce():
start = time.clock()
for i in range(m): # do it in a loop to make it take longer
    s = reduce(lambda x,y: x+y, X)
print "it look %f seconds to use reduce()"%(time.clock()-start)
print s
# this is a LOT slower.

# now my suggestion:
start = time.clock()
for i in range(m): # do it in loop to make it take longer
    s = sum(X.copy().astype(Int))
print "it look %f seconds to use Numeric.sum, with a
typecast"%(time.clock()-start)
print s

it look 0.090000 seconds to use Numeric.sum
14
it look 11.390000 seconds to use reduce()
63494670
it look 0.470000 seconds to use Numeric.sum, with a typecast
63494670

What I suggested is about 5 times slower than what he wants, and uses
four times as much memory.

The reduce() method doesn't use the memory, but takes 126 times as long.
If speed matters in this application, that is a BIG difference!

-Chris


-- 
Christopher Barker,
Ph.D.                                                           
ChrisHBarker at home.net                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------