[Numpy-discussion] Catching out-of-memory error before it happens

Francesc Alted francesc at continuum.io
Fri Jan 24 10:33:45 EST 2014


Yeah, numexpr is pretty cool for avoiding temporaries in an easy way:

https://github.com/pydata/numexpr

Francesc

El 24/01/14 16:30, Nathaniel Smith ha escrit:
>
> There is no reliable way to predict how much memory an arbitrary numpy 
> operation will need, no. However, in most cases the main memory cost 
> will be simply the need to store the input and output arrays; for 
> large arrays, all other allocations should be negligible.
>
> The most effective way to avoid running out of memory, therefore, is 
> to avoid creating temporary arrays, by using only in-place operations.
>
> E.g., if a and b each require N bytes of ram, then memory requirements 
> (roughly).
>
> c = a + b: 3N
> c = a + 2*b: 4N
> a += b: 2N
> np.add(a, b, out=a): 2N
> b *= 2; a += b: 2N
>
> Note that simply loading a and b requires 2N memory, so the latter 
> code samples are near-optimal.
>
> Of course some calculations do require the use of temporary storage 
> space...
>
> -n
>
> On 24 Jan 2014 15:19, "Dinesh Vadhia" <dineshbvadhia at hotmail.com 
> <mailto:dineshbvadhia at hotmail.com>> wrote:
>
>     I want to write a general exception handler to warn if too much
>     data is being loaded for the ram size in a machine for a
>     successful numpy array operation to take place.  For example, the
>     program multiplies two floating point arrays A and B which are
>     populated with loadtext.  While the data is being loaded, want to
>     continuously check that the data volume doesn't pass a threshold
>     that will cause on out-of-memory error during the A*B operation.
>     The known variables are the amount of memory available, data type
>     (floats in this case) and the numpy array operation to be
>     performed. It seems this requires knowledge of the internal memory
>     requirements of each numpy operation.  For sake of simplicity, can
>     ignore other memory needs of program.  Is this possible?
>
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


-- 
Francesc Alted

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140124/e7d3e021/attachment.html>


More information about the NumPy-Discussion mailing list