[Numpy-discussion] Catching out-of-memory error before it happens
Francesc Alted
francesc at continuum.io
Fri Jan 24 10:33:45 EST 2014
Yeah, numexpr is pretty cool for avoiding temporaries in an easy way:
https://github.com/pydata/numexpr
Francesc
El 24/01/14 16:30, Nathaniel Smith ha escrit:
>
> There is no reliable way to predict how much memory an arbitrary numpy
> operation will need, no. However, in most cases the main memory cost
> will be simply the need to store the input and output arrays; for
> large arrays, all other allocations should be negligible.
>
> The most effective way to avoid running out of memory, therefore, is
> to avoid creating temporary arrays, by using only in-place operations.
>
> E.g., if a and b each require N bytes of ram, then memory requirements
> (roughly).
>
> c = a + b: 3N
> c = a + 2*b: 4N
> a += b: 2N
> np.add(a, b, out=a): 2N
> b *= 2; a += b: 2N
>
> Note that simply loading a and b requires 2N memory, so the latter
> code samples are near-optimal.
>
> Of course some calculations do require the use of temporary storage
> space...
>
> -n
>
> On 24 Jan 2014 15:19, "Dinesh Vadhia" <dineshbvadhia at hotmail.com
> <mailto:dineshbvadhia at hotmail.com>> wrote:
>
> I want to write a general exception handler to warn if too much
> data is being loaded for the ram size in a machine for a
> successful numpy array operation to take place. For example, the
> program multiplies two floating point arrays A and B which are
> populated with loadtext. While the data is being loaded, want to
> continuously check that the data volume doesn't pass a threshold
> that will cause on out-of-memory error during the A*B operation.
> The known variables are the amount of memory available, data type
> (floats in this case) and the numpy array operation to be
> performed. It seems this requires knowledge of the internal memory
> requirements of each numpy operation. For sake of simplicity, can
> ignore other memory needs of program. Is this possible?
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Francesc Alted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140124/e7d3e021/attachment.html>
More information about the NumPy-Discussion
mailing list