[Numpy-discussion] Catching out-of-memory error before it happens

Nathaniel Smith njs at pobox.com
Fri Jan 24 10:30:34 EST 2014

There is no reliable way to predict how much memory an arbitrary numpy
operation will need, no. However, in most cases the main memory cost will
be simply the need to store the input and output arrays; for large arrays,
all other allocations should be negligible.

The most effective way to avoid running out of memory, therefore, is to
avoid creating temporary arrays, by using only in-place operations.

E.g., if a and b each require N bytes of ram, then memory requirements

c = a + b: 3N
c = a + 2*b: 4N
a += b: 2N
np.add(a, b, out=a): 2N
b *= 2; a += b: 2N

Note that simply loading a and b requires 2N memory, so the latter code
samples are near-optimal.

Of course some calculations do require the use of temporary storage space...

On 24 Jan 2014 15:19, "Dinesh Vadhia" <dineshbvadhia at hotmail.com> wrote:

>  I want to write a general exception handler to warn if too much data is
> being loaded for the ram size in a machine for a successful numpy array
> operation to take place.  For example, the program multiplies two floating
> point arrays A and B which are populated with loadtext.  While the data is
> being loaded, want to continuously check that the data volume doesn't pass
> a threshold that will cause on out-of-memory error during the A*B
> operation.  The known variables are the amount of memory available, data
> type (floats in this case) and the numpy array operation to be performed.
> It seems this requires knowledge of the internal memory requirements of
> each numpy operation.  For sake of simplicity, can ignore other memory
> needs of program.  Is this possible?
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140124/80e473a2/attachment.html>

More information about the NumPy-Discussion mailing list