Evan Jones wrote:
Some posts to various lists [1] have stated that this is not a real problem because virtual memory takes care of it. This is fair if you are talking about a couple megabytes. In my case, I'm talking about ~700 MB of wasted RAM, which is a problem.
This is not true. The RAM is not wasted. As you explain later, the pages will be swapped out to swap space, making the RAM available again for other tasks.
First, this is wasting space which could be used for disk cache, which would improve the performance of my system.
And indeed, this is what the operating system does for you: free the memory (by swapping it out), then using the memory for disk cache, thus improving performance of your system.
Second, when the system decides to swap out the pages that haven't been used for a while, they are dirty and must be written to swap.
That is true.
If Python ever wants to use them again, they will be brought it from swap.
Yes. However, your assumption is that Python never wants to use them again, because the peek memory consumption is only local. On the design memory management systems, there is the notion of a working set. Python will have a relatively constant working set over short periods of time, and current operating systems will manage to keep the working set in memory if the system has sufficient memory in the first place. As the working set grows or shrinks, pages get swapped in and out. As Tim explains, this is really hard to avoid.
This is much worse than informing the system that the pages can be discarded, and allocating them again later.
Unfortunately, as Tim explains, there is no way to reliably "inform" the system. free(3) may or may not be taken as such information. Regards, Martin