Maickel Pandie wrote:

I had research about how fast MTA (postfix) can deliver a lot of email from mailman. The one that I want to change is queue process method from FIFO to size based(sum of domain in the list x size of email), I have a list that contain the sum of domain each mailing list. I want Mailman prefer sending email that have less domain destination and small size of email.

Can you tell me what file I have to edit so mailman can decide what email is thrown to postfix first?

You would need to modify the Switchboard.files() method in module Mailman/Queue/Switchboard.py.

Think hard before you move away from FIFO queue processing. If your out queue is not backlogged, it will essentially be processed FIFO in any case because each time you look there will only be one or two entries to process. If it is backlogged and you process in any but FIFO order, you run the risk of leaving non-preferred entries unprocessed for very long times.

Also, your specific strategy would penalize larger (perhaps more popular) lists by delaying posts to those lists in favor of those to smaller (perhaps obscure) lists. Is this really what you want to do?

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Thanks Mark and Ian for your suggestion, I want ask again,

1. Is queue runners like Incomingqueue runner so I have to make more Incomingqueue runner?
2. How can I made more queue runners?
3. Can I make more queue pipe so incoming email is deliver to pipe depend on their size and domain destination? 4 If I bombard my mailing list server using email stress test can I make a lot of waiting process in queue?

Thanks

---------- Forwarded message ---------- From: Ian Eiloart <iane@sussex.ac.uk> Date: 2008/6/4 Subject: Re: [Mailman-Developers] How To change queue process method To: Mark Sapiro <mark@msapiro.net>, Maickel Pandie <maickel.pandie@gmail.com>, mailman-developers@python.org

--On 3 June 2008 11:51:31 -0700 Mark Sapiro <mark@msapiro.net> wrote:

Maickel Pandie wrote:

...

I had research about how fast MTA (postfix) can deliver a lot of email from mailman. The one that I want to change is queue process method from FIFO to size based(sum of domain in the list x size of email), I have a list that contain the sum of domain each mailing list. I want Mailman prefer sending email that have less domain destination and small size of email.

Can you tell me what file I have to edit so mailman can decide what email is thrown to postfix first?

You would need to modify the Switchboard.files() method in module Mailman/Queue/Switchboard.py.

Think hard before you move away from FIFO queue processing. If your out queue is not backlogged, it will essentially be processed FIFO in any case because each time you look there will only be one or two entries to process. If it is backlogged and you process in any but FIFO order, you run the risk of leaving non-preferred entries unprocessed for very long times.

Also, your specific strategy would penalize larger (perhaps more popular) lists by delaying posts to those lists in favor of those to smaller (perhaps obscure) lists. Is this really what you want to do?

A better solution might be to configure Mailman to use more queue runners. We saw a massive performance increase when we did that, because a single large delivery would not hold up other quicker deliveries.

-- Ian Eiloart IT Services, University of Sussex x3148

--On 6 June 2008 15:23:57 +0700 Maickel Pandie <maickel.pandie@gmail.com> wrote:

Thanks Mark and Ian for your suggestion, I want ask again,

1. Is queue runners like Incomingqueue runner so I have to make more Incomingqueue runner? yes
2. How can I made more queue runners? For example, put this in Mailman/mm_cfg.py :

QRUNNERS = [ ('ArchRunner', 4), # messages for the archiver ('BounceRunner', 4), # for processing the qfile/bounces directory ('CommandRunner', 4), # commands and bounces from the outside world ('IncomingRunner', 4), # posts from the outside world ('NewsRunner', 4), # outgoing messages to the nntpd ('OutgoingRunner', 4), # outgoing messages to the smtpd ('VirginRunner', 4), # internally crafted (virgin birth) messages ('RetryRunner', 4), # retry temporarily failed deliveries ]

the number must be a power of 2, so if you want to go higher, try 8 or 16. Perhaps it's only required for OutgoingRunner and RetryRunner.

The performance improvement that I saw after doing this was so huge that I can't believe it isn't standard. Essentially it means that one big delivery job doesn't block the rest of the lists.

3. Can I make more queue pipe so incoming email is deliver to pipe depend on their size and domain destination?

I don't know. It would depend on your MTA, and you haven't said what that is. If it isn't Exim, I can't help.

4 If I bombard my mailing list server using email stress test can I make a lot of waiting process in queue?

Probably not without queueing lots of mail to third party domains. You'd want to be very careful not to spam people.

Thanks

---------- Forwarded message ---------- From: Ian Eiloart <iane@sussex.ac.uk> Date: 2008/6/4 Subject: Re: [Mailman-Developers] How To change queue process method To: Mark Sapiro <mark@msapiro.net>, Maickel Pandie <maickel.pandie@gmail.com>, mailman-developers@python.org

--On 3 June 2008 11:51:31 -0700 Mark Sapiro <mark@msapiro.net> wrote:

...

Maickel Pandie wrote:

...

I had research about how fast MTA (postfix) can deliver a lot of email from mailman. The one that I want to change is queue process method from FIFO to size based(sum of domain in the list x size of email), I have a list that contain the sum of domain each mailing list. I want Mailman prefer sending email that have less domain destination and small size of email.

Can you tell me what file I have to edit so mailman can decide what email is thrown to postfix first?

You would need to modify the Switchboard.files() method in module Mailman/Queue/Switchboard.py.

Think hard before you move away from FIFO queue processing. If your out queue is not backlogged, it will essentially be processed FIFO in any case because each time you look there will only be one or two entries to process. If it is backlogged and you process in any but FIFO order, you run the risk of leaving non-preferred entries unprocessed for very long times.

Also, your specific strategy would penalize larger (perhaps more popular) lists by delaying posts to those lists in favor of those to smaller (perhaps obscure) lists. Is this really what you want to do?

A better solution might be to configure Mailman to use more queue runners. We saw a massive performance increase when we did that, because a single large delivery would not hold up other quicker deliveries.

-- Ian Eiloart IT Services, University of Sussex x3148

---

Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/iane%40sussex.a c.uk

Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

-- Ian Eiloart IT Services, University of Sussex x3148

Maickel Pandie wrote:

My MTA is postfix

Mark Sapiro wrote: "You could also just stop Mailman, deliver a lot of mail and then start Mailman." Thanks I will try that

Also, in the Mailman distribution there is tests/fblast.py which delivers lots of mail to Mailman.

Question again:

1. am I right that all of new email who comes to the server are queued in incoming queue by incomingrunner and the step is like this? "new email"----->Incoming queue----->Outgoing queue----> MTA queue ^ ^ ^ Incomingrunner Outgoingrunner ?

No. New mail is piped by the MTA to the mail/mailman wrapper which invokes one of the scripts in the scripts directory to put the message in the appropriate queue. Mail to the list posting address invokes the scripts/post script which queues the mail in the 'in' queue.

IncomingRunner processes the 'in' queue and in the normal case will queue a message to be archived in the 'archive' queue and a message to be delivered in the 'out' queue. ArchRunner processes the 'archive' queue and adds messages to the archive. OutgoingRunner processes the 'out' queue and delivers messages via SMTP to the MTA.

2. in what queue to most posibility a long email queue will happen if I do what Mark Sapiro suggested?

Messages will be queued in the 'in' queue and will remain there until you start IncomingRunner.

3. can I maintain what email is thrown first between Incoming queue to Outgoing queue?

It is strictly FIFO.

4. what runner should I edit if I want to make number 3 happen?

As mentioned before in this thread <http://mail.python.org/pipermail/mailman-developers/2008-June/020207.html> you would not edit a qrunner. You would edit the Switchboard.files() method in module Mailman/Queue/Switchboard.py to return the files in the order you want.

5. am I right that switchboard.py is always use when there are email moving from one queue to another queue to maintain FIFO in all of queue in mailman so if I want to change the FIFO I just have to edit switcboard.py?

Yes. The Switchboard class does all queueing and dequeueing and it is the files() method which returns a list of files in a queue in the order that they will be processed.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Hi...

I have some problem here and I didn't know how to solve it. I sent 1 message to milis_01@milis.com (listname) and I record what Switchboard.files() return to Runner.py:

1213621876.0387881+53028ae65697f5b1c06e9cf0f3b04ae1bb2ecfb1 1213621876.0387881+ad9a762eff62186e94d26fda95865243b6dc4069 1213621876.0387881+e7efb136c33f77e0674e03512860aadb23a2d605

From Switchboard.enqueue() and Switchboard.files() I knew that 1213621876.0387881 is the rcvtime

What I asking are:

1. Why the digest (53028ae65697f5b1c06e9cf0f3b04ae1bb2ecfb1, ad9a762eff62186e94d26fda95865243b6dc4069, e7efb136c33f77e0674e03512860aadb23a2d605) are changing for same message?

2. In Switchboard.files() Can I get listname from the digest/filebase? How?

2008/6/11 Maickel Pandie <maickel.pandie@gmail.com>:

Ok, many thanks I will try to edit switcboard.py first. If there any result I have, I will post the result... and if I get any problem I will ask for help again :) Thanks for your guidance

---------- Forwarded message ---------- From: Mark Sapiro <mark@msapiro.net> Date: 2008/6/10 Subject: Re: [Mailman-Developers] Fwd: Fwd: How To change queue process method To: Maickel Pandie <maickel.pandie@gmail.com>, mailman-developers@python.org

Maickel Pandie wrote:

...

My MTA is postfix

Mark Sapiro wrote: "You could also just stop Mailman, deliver a lot of mail and then start Mailman." Thanks I will try that

Also, in the Mailman distribution there is tests/fblast.py which delivers lots of mail to Mailman.

...

Question again:

1. am I right that all of new email who comes to the server are queued in incoming queue by incomingrunner and the step is like this? "new email"----->Incoming queue----->Outgoing queue----> MTA queue ^ ^ ^ Incomingrunner Outgoingrunner ?

No. New mail is piped by the MTA to the mail/mailman wrapper which invokes one of the scripts in the scripts directory to put the message in the appropriate queue. Mail to the list posting address invokes the scripts/post script which queues the mail in the 'in' queue.

IncomingRunner processes the 'in' queue and in the normal case will queue a message to be archived in the 'archive' queue and a message to be delivered in the 'out' queue. ArchRunner processes the 'archive' queue and adds messages to the archive. OutgoingRunner processes the 'out' queue and delivers messages via SMTP to the MTA.

...

2. in what queue to most posibility a long email queue will happen if I do what Mark Sapiro suggested?

Messages will be queued in the 'in' queue and will remain there until you start IncomingRunner.

...

3. can I maintain what email is thrown first between Incoming queue to Outgoing queue?

It is strictly FIFO.

...

4. what runner should I edit if I want to make number 3 happen?

As mentioned before in this thread <http://mail.python.org/pipermail/mailman-developers/2008-June/020207.html> you would not edit a qrunner. You would edit the Switchboard.files() method in module Mailman/Queue/Switchboard.py to return the files in the order you want.

...

5. am I right that switchboard.py is always use when there are email moving from one queue to another queue to maintain FIFO in all of queue in mailman so if I want to change the FIFO I just have to edit switcboard.py?

Yes. The Switchboard class does all queueing and dequeueing and it is the files() method which returns a list of files in a queue in the order that they will be processed.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan