[Numpy-discussion] Summation of large float32/float64 arrays

[Christoph Gohlke]

On 5/21/2010 2:30 PM, Alan G Isaac wrote:
> On 5/21/2010 4:13 PM, Matthew Turk wrote:
>> a1 = numpy.random.random((512,512,512)).astype("float32")
>
>
> This consistently gives me a "MemoryError".
> I believe I have plenty of physical memory.
> (That should require about 1.3G during creation, right?
> I have 8G.) It seems I'm hitting some kind of 1G
> memory use limit.How to think about this?
>
> I can create the initial 64 bit array no problem.
> However I cannot create the second 32 bit array,
> despite having plenty of physical memory.
> The copy method also fails; or even creating a
> second array the same size fails, unless I first
> delete `a1`.
>
> I realize this is probably a naive and operating system
> dependent question.  Apologies if it is off topic.
>
> Thanks,
> Alan Isaac
> (running 32bit Python 2.6.5 on 64bit Vista;
> NumPy version import 1.4.1rc2)
>


The MemoryError is not not unexpected. First, the 32-bit Python 
interpreter can only use 2 GB of your memory. Second, numpy arrays need 
a contiguous, un-fragmented, range of memory to be available within that 
2 GB address space. In this example there is no contiguous 512 MB space 
left after creating the 1 GB array. Practically it does not seem 
possible to allocate a single array larger than about 1.3 GB on win32. 
The solution is to use a 64-bit version of Python and numpy.

Christoph