On Tue, Dec 27, 2005 at 03:28:30AM +0100, Jesus Cea wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Wow, Andrea Arcangeli, of Linux Kernel fame. What an honor :).
+ import gc + gc.collect()
Haha. I was just writing you about manual garbage collector invocations.
;) Thanks! I couldn't even think that it could have been the garbage collector because the thing was >1G large. I definitely expected any sane garbage collector to kick in at least once every time the task was doubling in size and certainly before the thing could grow to the gigabytes. Perhaps this is what should really happen. Every time the task doubles in size we could force a gc.collect(). The growth rate that should trigger a collect() could be a threshold tunable too (default None to be backwards compatible even though I think None is not a safe/sane default).
Manual collections can be costly, nevertheless. Since the manual collector is only needed to free cyclic references, I guess that Twisted people would rather prefer a patch to break cyclic references in Twisted code. For example, using "__del__" methods or "weakref".
You can easily see what objects and references keep the cycles alive using "gc.garbage" and "gc.get_referrers()". Usually a single "a.value=None" can do miracles :-). Weakref can be very very helpful, also.
a.value=None + weakrefs is what did the miracles indeed, but I did it in my code not in twisted code. twisted code has no cyclic references. All cyclic references were in my code sitting on top of twisted. It's me implementing the protocol. The a.value=None made the code a mess, the weakref complicated things too (especially because I had to hash one of those weak values, so I had to use weakref.ref and I had to change lots of code to add "()"). So I'm soooo glad to have backed out all those changes because they were unnecessary ;). Plus a.value = None is similar to freeing memory in a normal language without garbage collection, not very different from free(a.value). So I certainly prefer to use the more advanced features of the garbage collector, even if I have to invoke it by hand... ;)