[Mailman-Developers] re: Much cpu/memory load

[Dan Mick]

Dan Mick wrote:

>> Barry, wasn't there some Python instrumentation trick for using the GC 
>> to find unreferenced objects and complain, or invoking GC manually in 
>> a loop to try to alleviate such problems?....was it in the gc module?...

Adding this:

--- /usr/local/src/mailman/Mailman/Archiver/pipermail.py        Tue Oct 15 
20:22
:49 2002
+++ /export/home/mailman/Mailman/Archiver/pipermail.py  Fri Nov  1 12:48:47 
2002
@@ -9,6 +9,7 @@
  import cPickle as pickle
  from cStringIO import StringIO
  from string import lowercase
+import gc

  __version__ = '0.09 (Mailman edition)'
  VERSION = __version__
@@ -425,6 +426,7 @@
          self._update_thread_index(archive, arcdir)

      def _update_simple_index(self, hdr, archive, arcdir):
+       gc.set_debug(gc.DEBUG_LEAK)
          self.message("  " + hdr)
          self.type = hdr
          hdr = hdr.lower()
@@ -442,6 +444,7 @@
              else:
                  count = count + 1
                  self.write_index_entry(article)
+            sys.stderr.write("gc, number of uncollectables: %d\n" % 
gc.collect(
))
              msgid = self.database.next(archive, hdr)
          # Finish up this index
          self.write_index_footer()


prints a bunch of "gc, number of uncollectables: 0", and doesn't stop the
memory growth.  Note that the time between each round of brks() is high, 
too, as though there's a *lot* of userland work going on.  That process can 
take 70% of a CPU by itself while running.