[Mailman-Users] Problem with qrunner and too much incoming mail

[Marc MERLIN]

On Fri, Nov 03, 2000 at 12:24:47PM -0800, Chuq Von Rospach wrote:
> >We're currently playing with  MTAs to optimize this a bit,  but the real fix
> >is on the mailing list side.
> 
> what's your MTA? Sendmail? postfix?
 
sendmail, switching to exim (hence the optimization).
I'm talking about the exim folks to improve performance, but the real
problem is with qrunner, not exim.
 
> >- Forget about qrunner and switch back to direct delivery and queueing only
> >   when direct delivery fails.
> 
> the other problem with this is peak load problems. Instead of getting 
> nailed by a backlogged, you get nailed by trying to delivery all at 
> once and having your system croak. You need some way to manage 
> delivery to keep the peaks from killing you.

The machines in question can handle peak loads without problems.
Is direct SMTP delivery  (the option in Default.py) safe or  not (I was told
it  could  drop Emails  due  to  bugs in  the  handling  of certain  failure
conditions)?

> >- Have a multithreaded qrunner that processes 10 or 20 mails at once
> >   (talking to 10 or 20 instances of the MTa in parrallel)
> 
> something I was going to bring up -- since I'm now running Mailman on 
> a 2CPU system, it'd be nice to be able to run multiple qrunners. But 
> it actually has some fairly serious implications in the locking areas 
> as well, so it doesn't seem to be an easy fix. It's certainly not 
> something I'd try to load into the 2.0 timeframe, not if we're RC 
> candidate status.
 
Ok, so the current thread option in mailman doesn't do that then I take it.
 
> >- Other?
> 
> I think it's a combination of factors .The quick fix is tweaking the 
> MTA to be less sensitive. On my big server (50+ lists, 40,000+ 
> subscribers) I simply haven't see more than a couple of hour delay, 

I'm talking 10,000 lists, and loads of subscribers.
The mail part seems ok, mailman 1.1 was doing ok, it started failing when we
switched to the 2.0 tree with qrunner.

BTW, for us, a couple of hours delay isn't quite acceptable.

> >Now, if I  have X machines that mount /var/local/mailman,  they'd be able to
> >service all the lists (config.db would  get locked correctly), but I'd still
> >be stuck with only one queue runner because of the global lock.
> >That said,  I *could* have mailman/data  and mailman/qfiles be a  symlink to
> >somewhere on the local disk, and patch qrunner to put its lock in data.
> >This would allow for independant queue  runners, but shared list configs and
> >shared locks on the list configs themselves.
> >
> >Would that work?
> 
> I'm glad I'm not trying that on MY machine. I wish you luck...

:-)

Marc
-- 
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
  
Home page: http://marc.merlins.org/   |   Finger marc_f at merlins.org for PGP key