[Python-Dev] Changing pymalloc behaviour for long running processes

Gregory P. Smith greg at electricrain.com
Thu Oct 28 10:09:01 CEST 2004

On Tue, Oct 19, 2004 at 08:00:56PM +0200, "Martin v. L?wis" wrote:
> Evan Jones wrote:
> >Some posts to various lists [1] have stated that this is not a real  
> >problem because virtual memory takes care of it. This is fair if you  
> >are talking about a couple megabytes. In my case, I'm talking about  
> >~700 MB of wasted RAM, which is a problem. 
> This is not true. The RAM is not wasted. As you explain later, the
> pages will be swapped out to swap space, making the RAM available
> again for other tasks.
> >First, this is wasting space  
> >which could be used for disk cache, which would improve the performance  
> >of my system. 
> And indeed, this is what the operating system does for you: free the
> memory (by swapping it out), then using the memory for disk cache, thus
> improving performance of your system.

In the long run on a system the RAM may not be wasted once the OS
happens to have swapped it out but the address space is still used.
You're still consuming ~700 MB of your OS's total address space with
swapped garbage.  The fact that ultimately a lot of it ends up on disk
as swap is not nice to other processes wanting memory (and disk space
for oses using a dynamic swap).

That said, here's a workaround for avoiding permanent huge memory
consumption in known workloads:

fork() before doing the part that consumes a ton of memory.  afterwards
return the results, post huge memory consumption, via pipe to the waiting
parent process and exit the child so the parent can continue on not
consuming 700mb.

More information about the Python-Dev mailing list