[Numpy-discussion] Numpy doesn't use RAM

[Stanley Seibert]

In addition to what Sebastian said about memory fragmentation and OS limits
about memory allocations, I do think it will be hard to work with an array
that close to the memory limit in NumPy regardless.  Almost any operation
will need to make a temporary array and exceed your memory limit.  You
might want to look at Dask Array for a NumPy-like API for working with
chunked arrays that can be staged in and out of memory:

https://docs.dask.org/en/latest/array.html

As a bonus, Dask will also let you make better use of the large number of
CPU cores that you likely have in your 1.9 TB RAM system.  :)

On Tue, Mar 24, 2020 at 1:00 PM Keyvis Damptey <quantkeyvis at gmail.com>
wrote:

> Hi Numpy dev community,
>
> I'm keyvis, a statistical data scientist.
>
> I'm currently using numpy in python 3.8.2 64-bit for a clustering problem,
> on a machine with 1.9 TB RAM. When I try using np.zeros to create a 600,000
> by 600,000 matrix of dtype=np.float32 it says
> "Unable to allocate 1.31 TiB for an array with shape (600000, 600000) and
> data type float32"
>
> I used psutils to determine how much RAM python thinks it has access to
> and it return with 1.8 TB approx.
>
> Is there some way I can fix numpy to create these large arrays?
> Thanks for your time and consideration
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200324/02cbeb71/attachment.html>