On Oct 19, 2004, at 9:44, Nick Coghlan wrote:
I knew this discussion sounded familiar. . .
http://mail.python.org/pipermail/python-dev/2004-June/045403.html (and assorted replies)
Great! Thank you for the reference, this discussion is useful. Why is it that it is so hard to find stuff in mailing list archives? I spent about 5 hours poking around Google and manually browsing the archives, and didn't see that thread.
I'm not saying I *like* the unbounded lists. . . but there's a reason they're still like that (i.e. getting the memory usage down tends to take some of Python's speed with it - and there isn't exactly a lot of that to be spared!).
Of course, and this has to be carefully considered and carefully benchmarked as well. I don't think that a bounded free list approach is the answer, as it can be shown that any arbitrary, fixed limit is bad is some situations. Basically, I'm talking about including some cache management into Python. My idea is to add some instrumentation that monitors the size of the free lists over time. If the free lists are much larger than what is being used, go and free some of them. This will allow Python's memory usage to dynamically adjust to the workload, and should limit the overhead. There are two challenges to this idea: 1. How can we run such an occasional task? It should be called regularly, if Python is actively executing or not, just like a garbage collector thread would be. I don't think making this collection part of the normal call to allocate/free objects is a good idea, because that would add some expensive overhead in what needs to be a fast path for Python. 2. Can this be done with minimal overhead? The instrumentation needs to have near zero overhead, and the collection task needs to have small overhead. I haven't thought this out completely, but it seems to me that something better should be possible. Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso