- Mark Sapiro mark@msapiro.net wrote:
Stefan Förster wrote:
I don't think I'm qualified to discuss my concern (which involves VERP delivery using a specific MTAs VERP implementation and a highly I/O saturated mail server) on this list.
See http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.012.htp for some background.
I apologize, I should have found that entry myself. This sure makes up for an interesting reading. Was bandwith really that expensive back in 2001? :)
- is only partially correct. The purpose of grouping is to try to
minimize the number of transactions between the outgoing MTA and the remote MTA. E.g., if all the aol.com addresses are passed from mailman to the outgoing MTA in one SMTP transaction, then the MTA is able (if so configured, etc., ...) to deliver the message to aol.com in one transaction with the same recipients. It has nothing to do with trying to prevent blocking, as the chunks are delivered serially anyway.
I can fully understand the thoughts behind this. OTOH, as the FAQ entry you mentioned, if you do parallel deliveries to a single location, you get a greater throughput: "Cut network bandwidth but slow delivery to the larger domains." I remember a posting on this list about cuddling MTAs or trying to outsmart them ;-) [1]
In any case, if you have 'tuning' suggestions, either in general or for specific cases, we would be glad to receive them and either incorporate them in the code or add them to the FAQ as appropriate.
I don't know whether this is really a "tuning" suggestion. Let me please describe my specific setup in hopes that the reason for my request becomes clearer:
I recently had to take care of a server which was MX for a lagre company and their outgoing mail relay (common setup). This machine was doing fine until another server, dedicated to handle the companys many mailig lists, broke down and it had to take care of mailing list deliveries, too. The added load of permanent VERPed deliveries was totally killing the machine in terms of I/O capacity, a situation which could be resolved as of yesterday by letting the MTA do the VERPing.
Now, the reason this change was so effective was that the number of (disk) I/O operations was greatly reduced: Where a posting to a list with 25000 subscribers would inevitably lead to the creation of 25000 queue files (even more if message content and control information are kept in different files) before the change, only 25000/500 were created afterwards (or "should have been", see next paragraph). Additionally (but this has not really that much of an influence), a larger number of destinations is available much faster to the MTA, allowing it's queue manager to make decisions in a "more educated" way.
Since the server is still quite close to it's I/O limits, I would very much like to further reduce the number of I/O transactions. Therefore, I would like to see a configurable parameter, something like FLAT_RECIPIENT_CHUNKIFY: Right now, if mm_cfg.SMTP_MAX_RCPTS is not set <= 0, "chunkify" will always (i.e. for any real world recipient distribution) return more than one chunk of recipients, even if the total number of recipients is smaller than mm_cfg.SMTP_MAX_RCPTS. With the addition of the aforementioned "tunable", we could have chunkify simple split recipients into batches of mm_cfg.SMTP_MAX_RCPTS and don't do any groupings. This way, a delivery to 25000 recipients would really only involve (25000/mm_cfg.SMTP_MAX_RCPTS)*(number of queue files per message) file creation operations.
Let me please emphasize that I'm fully aware that I will need better disks or a new server, or get rid off VERP in the long run. I was just thinking that the addition of a tunable like that _might_ be a reasonable thing to propose if one absolutely wants to reduce the number of I/O operations during a "bulkdeliver" without having to set mm_cfg.SMTP_MAX_RCPTS to <= 0. But I think having one more option (and I don't think this is hard to code, even I could do that and I have read my first lines of Python on Friday evening) to adapt Mailman (or it's delivery strategy) to a local site's needs would not really hurt. If this option defaults to false, nobody would even notice it.
OTOH, it is one configuration parameter more, and one should not easily include these things to avoid unnecessary configuration complexity.
If you feel this is an unreasonable request, please do say so. If not, I'd be glad to get some feedback from the people on this list regarding "optimal" delivery strategies to the local MTA in this (or other) cases.
Cheers Stefan
[1] This was meant to be a joke. Just to be safe.
Stefan Förster http://www.incertum.net/ Public Key: 0xBBE2A9E9 "Wenn es sich um Geld handelt, gehört jeder der gleichen Religion an." -- Voltaire