[python-win32] MemoryError: can I use more?

[Dave Angel]

Greg Antal wrote:
> <div class="moz-text-flowed" style="font-family: -moz-fixed">Dear 
> Angelica:
>
> 2.2 GB is essentially the largest integer you express with 32 bits 
> (2^31), and therefore the largest quantity of memory for which you can 
> define an address. There are some tricks that really sophisticated 
> programmers can do that let you use more than that, but I don't know 
> how to do it. If you want to access more memory, you have to go to a 
> 64-bit machine and operating system.
>
> - Greg Antal
>
> Gregory W. Antal
> Senior Technical Advisor
> ATA Engineering, Inc.
> 11995 El Camino Real, Suite 200   
> San Diego, CA  92130
> www.ata-e.com
>
> greg.antal at ata-e.com
> 858-480-2072  (Phone)
> 858-792-8932  (Fax)
>
>
>
> Echavarria Gregory, Maria Angelica wrote, On 2/12/2010 4:22 PM:
>> Dear group:
>>
>> I am developing a program using Python 2.5.4 in windows 32 OS. The 
>> amount of data it works with is huge. I have managed to keep memory 
>> footprint low, but have found that, independent of the physical RAM 
>> of the machine, python always gives the MemoryError message when it 
>> has occupied exactly only 2.2 GB. I have tested this in 4 different 
>> machines, all with memory of 3 to 4 GB... I'm amazed.
>>
>> Could any of you please help me to figure out how to change that 
>> limit? I typed help(MemoryError) and it is a class itself, but that 
>> help told me nothing I can use...
>>
>> Thanks,
>> Angelica.
>> _______________________________________________
>> python-win32 mailing list
>> python-win32 at python.org
>> http://mail.python.org/mailman/listinfo/python-win32
>>   
>
>
It has little to do with physical RAM.  You'd hit the same limits with 
1gb of physical RAM, as long as you had a large enough swap file.

There are two limits on an individual application, the amount of swap 
space, and the amount of address space.  Swap space management depends 
on which "windows 32 OS" you're running, and how you have it 
configured.  I will have to assume you mean one of Windows XP, Windows 
Vista, or Windows 7.  And that swap space is shared among all the 
applications currently running.

But I expect you're bumping into a virtual address space limitation.  
Even if you could have a 50gb swapfile, that would raise the number of 
large applications you could run (very slowly), but not the size of any 
one application.

Your address space is 32 bits, or 4 gigabytes.  Every pointer in your 
program, or in the system has to point to a unique place, so there are 
only 4 gigabytes of virtual space.   By default Windows reserves the 
upper half of that for system DLL's, device stuff,  and working space.  
It also reserves the bottom meg or so for mostly historical reasons, and 
to catch null pointer errors.  So you only get 2gig of space
 for each process.

Two ways of extending that.  There's something called PAE (and AWE ), 
which allows you to read and write things "above" 4gb, but I don't know 
anything about how it may be accessible from Python.  Certainly you 
couldn't store ordinary python objects there.  Think of it as a 
(probably) fast disk partition with special read/write methods.

But the one you might want is a boot.ini option that tells the OS to 
only reserve 1gb for itself, and leave 3gb for user space.  But there 
are tradeoffs, including the need to modify an application's executable 
to take advantage of it.  And the whole system may run substantially 
slower, even when your're extended app isn't running.  See links:
   http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
   
http://blogs.technet.com/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx

Bottom line?  If you really need more than 2gig of user space, probably 
you should go to a 64 bit OS.

DaveA