[PYTHON MATRIX-SIG] How to Avoid the Mega-malloc?

[Nigel O'Brian]

David Ascher <da@maigret.cog.brown.edu> writes:

>> PS: I can't wait for spring -- I'm assuming it will help the tone on the
>>     main list, since nothing else seems to do...

Sorry, down here it's Fall, so I can't help you much.

Suppose I write a NumPy formula like

  A = 0.1*B + 0.2*C

where A, B and C are, say, arrays of 1 million floats. This causes a
modest 16Mb Linux box to swap like crazy. Depending what else is
happening, it may just run out of memory altogether. I'm guessing that
memory is being claimed for (perhaps) several intermediate results.

But thanks to the ufuncs' optional 3rd argument I can do something like

  add(multiply(0.1,B,B),multiply(0.2,C,C),B)

leaving the answer in B (if I don't mind trashing B and C).  I guess
this avoids any claims on memory. But it seems like there is one place
where this trick doesn't work -- namely operations based on the dot
product, such as matrix multiplication.

To get around this I wrote a rather specialised C extension to do
matrix multiplication in place (and only on certain types of
array). This saved a lot of memory and produced something like a 2x
speedup over standard NumPy on more modestly sized arrays.

In fact I would gladly give up the 2x speedup to keep the uniformity
and generality of standard NumPy. Any chance of an optional third
argument to `dot' to receive the result? Or is there some other trick
that I can use to multiply big arrays without a malloc?

Nigel
--------------------------------------------------------------------
Nigel O'Brian                             obrian_n@maths.usyd.edu.au
School of Mathematics                     Ph: (02)-9351-4083
University of Sydney                      FAX: (02)-9351-4534                
Sydney NSW 2006, Australia









_______________
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________