MemoryError, can I use more?

[Christian Heimes]

Echavarria Gregory, Maria Angelica wrote:
> Dear group:
> 
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.

The amount of RAM in your boxes is unrelated to your issue. A 32bit
process is able to address only 4GB of virtual memory because it has no
way to access memory addresses beyond the 32bit border. Large amount of
the address space are reserved for multiple purposes like error checking
(e.g. the NULL pointer), the entry point of the program, its stack and
address space for reallocating dynamic linked libraries. On modern Linux
systems the space for the heap is about 2.6 to 2.7 GB large. The heap is
the memory segment used by malloc() and friends. It looks like the
memory segment for the heap is slightly smaller on Windows.

Virtual memory isn't the same as resident memory. A program may only
occupy 100 MB in your RAM (resident memory) but it may use 1.5 GB of
virtual memory. Virtual memory takes memory fragmentation and unused
pages into account. If you malloc() 1 GB memory but just file up the
first byte your program requires 1 GB of virtual memory but only a few
KB resident memory. At least that's the story on Linux and several other
Unix-like OSes. I'm not sure how Windows deals with memory.

If your program needs more than 2 GB of RAM you should switch to a 64bit
OS and a 64bit version of Python. Windows AMD64 builds for Python are
available on python.

Christian