BounceRunner eating up alot of memory
Hi,
after a few more tests, I am confronted to a problem: the BounceRunner seems to load the list in its memory when it receives a bounce (in order to process that bounce). But it does this: 1- regardless of whether it's going to need the list or not 2- without releasing the memory afterwards
I can live with 1-, but 2- can be a problem if you have several big lists. On my installation currently the BounceRunner has 92Mb of RAM. I don't mind it taking whatever it needs, but it should release it if I don't want to spend too much time swapping memory to disk.
All in all, I'm really starting to wonder why the list settings' data and the list members' data are mixed in one single pck file, and not in two (or more) separate files. When you need only to check the list settings, it would be so much faster to load a 1k file than to process a 77Mb file? In many situations it seems that would make Mailman much faster and have a smaller mem footprint!
-- Fil
"F" == Fil <fil@rezo.net> writes:
F> after a few more tests, I am confronted to a problem: the F> BounceRunner seems to load the list in its memory when it F> receives a bounce (in order to process that bounce). But it F> does this: 1- regardless of whether it's going to need the list F> or not 2- without releasing the memory afterwards F> I can live with 1-, but 2- can be a problem if you have several F> big lists. For 1) It's pretty much going to need the list object, but perhaps you mean the list lock? I'm not going to worry about that. For 2) That's a valid point. Please try the following patch (untested). I'll try to test it and if it looks good, I'll commit this change tomorrow. F> All in all, I'm really starting to wonder why the list F> settings' data and the list members' data are mixed in one F> single pck file, and not in two (or more) separate files. When F> you need only to check the list settings, it would be so much F> faster to load a 1k file than to process a 77Mb file? In many F> situations it seems that would make Mailman much faster and F> have a smaller mem footprint! I know it's bad, but at least the situation isn't any worse than it was for MM2.0. Fixing this is off the table for MM2.1, because it's long overdue and a change like this would be too disruptive. Fixing this is (a large part of) what MM3.0 is all about. -Barry Index: BounceRunner.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/Queue/BounceRunner.py,v retrieving revision 2.9 diff -u -r2.9 BounceRunner.py --- BounceRunner.py 7 Mar 2002 22:30:04 -0000 2.9 +++ BounceRunner.py 11 Mar 2002 04:51:36 -0000 @@ -31,6 +31,7 @@ class BounceRunner(Runner): QDIR = mm_cfg.BOUNCEQUEUE_DIR + CACHELISTS = 0 def _dispose(self, mlist, msg, msgdata): outq = get_switchboard(mm_cfg.OUTQUEUE_DIR) Index: Runner.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/Queue/Runner.py,v retrieving revision 2.17 diff -u -r2.17 Runner.py --- Runner.py 11 Mar 2002 04:21:33 -0000 2.17 +++ Runner.py 11 Mar 2002 04:51:37 -0000 @@ -34,9 +34,11 @@ class Runner: - def __init__(self, slice=None, numslices=1, cachelists=1): + CACHELISTS = 1 + + def __init__(self, slice=None, numslices=1): self._kids = {} - self._cachelists = cachelists + self._cachelists = self.CACHELISTS # Create our own switchboard. Don't use the switchboard cache because # we want to provide slice and numslice arguments. self._switchboard = Switchboard(self.QDIR, slice, numslices)
"BAW" == Barry A Warsaw <barry@zope.com> writes:
BAW> For 2) That's a valid point. Please try the following patch
BAW> (untested). I'll try to test it and if it looks good, I'll
BAW> commit this change tomorrow.If you apply the patch, you'll also have to fix the __init__() method of the OutgoingRunner. Remove the cachelists argument.
-Barry
@ Barry A. Warsaw <barry@zope.com> :
For 2) That's a valid point. Please try the following patch (untested). I'll try to test it and if it looks good, I'll commit this change tomorrow.
There must be something missing:
Traceback (most recent call last): File "/home/mailman/bin/qrunner", line 265, in ? main() File "/home/mailman/bin/qrunner", line 212, in main qrunner = make_qrunner(*runners[0]) File "/home/mailman/bin/qrunner", line 118, in make_qrunner qrunner = qrclass(slice, range) File "/home/mailman/Mailman/Queue/OutgoingRunner.py", line 42, in __init__ Runner.__init__(self, slice, numslices) File "/home/mailman/Mailman/Queue/Runner.py", line 43, in __init__ self._cachelists = CACHELISTS NameError: global name 'CACHELISTS' is not defined
-- Fil
participants (2)
-
barry＠zope.com -
Fil