[SciPy-User] Problem with handling big matrices with Windows

[Antonelli Maria Rosaria]

Hello, 

I can np.memmap of that size, but when I ask to assign a zeros matrix of
the same size to the np.memmap, it gives the same error :
fp = np.memmap(filename, dtype='float, mode ='w+', shape(80, 100, 384,
285))
fp = zeros((80, 100, 384, 285)).

Best, 
Rosa 

On 6/11/14 11:32 AM, "Matthieu Brucher" <matthieu.brucher at gmail.com> wrote:

>Yes, but I don't think that would change anything. It is still the
>same OS that allocates memory in the end.
>
>Cheers,
>
>2014-06-11 10:26 GMT+01:00 Antonelli Maria Rosaria
><maria-rosaria.antonelli at curie.fr>:
>> Hello,
>> Thanks for answering.
>> I do have only one matrix that big and some others 80*285*384.
>> Have you ever heard about numpy binaries by Christoph Gohlke ?
>>
>> Best,
>> Rosa
>>
>> On 6/11/14 11:20 AM, "Matthieu Brucher" <matthieu.brucher at gmail.com>
>>wrote:
>>
>>>How many such matrices do you have? What is the content? Even if you
>>>could allocate up to 2**64 bytes with a 64bits app, it is not possible
>>>to do so. You are limited by your RAM and the size of your swap file.
>>>If the OS can find enough memory, it will fail with the error message
>>>you have.
>>>Perhaps you should use memmap to map your saved array on disk, instead
>>>of having it in memory.
>>>
>>>Cheers,
>>>
>>>Matthieu
>>>
>>>2014-06-11 11:17 GMT+02:00 Antonelli Maria Rosaria
>>><maria-rosaria.antonelli at curie.fr>:
>>>> Sorry I have to correct my self.
>>>> I already have installed Anaconda 64 bitŠ
>>>> Best regards,
>>>> Rosa
>>>>
>>>> On 6/11/14 11:13 AM, "Matthieu Brucher" <matthieu.brucher at gmail.com>
>>>>wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>Your arrays have more than 800 millions entries, so even with floats
>>>>>or int32, that means more than 3GB, meaning only one would fit in the
>>>>>process memory.
>>>>>You _have_ to switch to a 64bits Python and hope you have enough RAM
>>>>>as
>>>>>well.
>>>>>
>>>>>Cheers,
>>>>>
>>>>>Matthieu
>>>>>
>>>>>2014-06-11 10:08 GMT+01:00 Antonelli Maria Rosaria
>>>>><maria-rosaria.antonelli at curie.fr>:
>>>>>> Hello,
>>>>>>
>>>>>> I am working with Windows 7, 64bits and I have 8G of Ram.
>>>>>> I have installed Anaconda 32bits and I get "memory" error when
>>>>>>handling
>>>>>> matrices of 80*100*285*384.
>>>>>> Do installing Anaconda 64bits would solve this problem ?
>>>>>> Can I install Anaconda 64 bits without uninstalling Anaconda 32bits
>>>>>>?
>>>>>>
>>>>>> Best regards,
>>>>>> Rosa
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>Information System Engineer, Ph.D.
>>>>>Blog: http://matt.eifelle.com
>>>>>LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>>>Music band: http://liliejay.com/
>>>>>_______________________________________________
>>>>>SciPy-User mailing list
>>>>>SciPy-User at scipy.org
>>>>>http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>>
>>>--
>>>Information System Engineer, Ph.D.
>>>Blog: http://matt.eifelle.com
>>>LinkedIn: http://www.linkedin.com/in/matthieubrucher
>>>Music band: http://liliejay.com/
>>>_______________________________________________
>>>SciPy-User mailing list
>>>SciPy-User at scipy.org
>>>http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>-- 
>Information System Engineer, Ph.D.
>Blog: http://matt.eifelle.com
>LinkedIn: http://www.linkedin.com/in/matthieubrucher
>Music band: http://liliejay.com/
>_______________________________________________
>SciPy-User mailing list
>SciPy-User at scipy.org
>http://mail.scipy.org/mailman/listinfo/scipy-user