Queue not releasing memory
Steven H. Rogers
shrogers at ionet.net
Tue Sep 21 20:26:30 EDT 1999
Nathan Clegg wrote:
>
> On 12-Sep-99 Tim Peters wrote:
> > Python has no control over when your OS decides that released memory is
> > available to something else. If you name your OS, perhaps someone will
> > know
> > how to influence it reliably.
>
> The results I described were under Linux, glibc 2.1.1. The target
> platform for this application, however, is Solaris 2.7, which is worse.
> Under Solaris, the memory is not reclaimed when the queue is deleted. The
> entire python process must terminated. Not excited about this. I just
> did a simple malloc/free test with the exact same results. Bad Solaris.
>
> > By the way, look at that code carefully: the default Queue
> > implementation
> > wasn't meant to handle memory-busting amounts of data! That "del
> > self.queue[0]" line makes a physical copy of the entire data remaining
> > every
> > time.
>
> I do need the thread-safety of the Queue class. The large size
> requirement is only an extreme situation, which is why it is important
> that it backs down afterward. Perhaps I need to take some pressure off
> Queue and move more quickly to my other lines of defense.
>
> Is there no way to remove an element from a list without making a copy?
> append() doesn't need a copy. I was under the impression that lists were
> internally represented by a more dynamic form.
>
> Thanks for the information. I will reevaluate my plan of attack.
>
> ----------------------------------
> Nathan Clegg
> nathan at islanddata.com
I believe that Solaris tends to consume whatever physical memory is
available, releasing it as needed for new demands. Memory utilization
usually appears to be high under this scenario.
-- Steve
More information about the Python-list
mailing list