[SciPy-User] Strange memory limits

[Chris Weisiger]

(This is unrelated to my earlier question about 2D data slicing)

We have a 32-bit Windows program that has Python bindings which do most of
the program logic, reserving the C++ side for heavy lifting. This program
needs to reserve buffers of memory to accept incoming image data from our
different cameras -- it waits until it has received an image from all active
cameras, then saves the image to disk, repeat until all images are in. So
the Python side uses numpy to allocate a block of memory, then hands it off
to the C++ side where images are written to it and then later stored.
Ordinarily all of our cameras are operating in sync so the delay between the
first and last cameras is small, so we can keep the memory buffer small. I'm
working on a modified data collection mode where each camera does a lengthy
independent sequence, though, requiring me to either rewrite the data saving
system or simply increase the buffer size.

Increasing the buffer size works just fine until I try to allocate about a
3x735x512x512 array (camera/Z/X/Y) of 16-bit ints, at which point I get a
MemoryError. This is only a bit over 1GB worth of memory (out of 12GB on the
computer), and according to Windows' Task Manager the program was only using
about 100MB before I tried the allocation -- of course, I've no idea how the
Task Manager maps to how much RAM I've actually requested. So that's a bit
strange. I ought to have 4GB worth of space (or at the very least 3GB),
which is more than enough for what I need.

Short of firing up a memory debugger, any suggestions for tracking down big
allocations? Numpy *should* be our only major offender here aside from the
C++ portion of the program, which is small enough for me to examine by hand.
Would it be reasonable to expect to see this problem go away if we rebuilt
as a 64-bit program with 64-bit numpy et al?

Thanks for your time.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20110328/fd689159/attachment.html>