Mailman 3 October 2003 - Mailman-Developers

Requirements for a new archiver
by Iain Bapty Nov. 6, 2003

Nov. 6, 2003

Hi, For those of you that don't know I am currently working on a archive component for Mailman as part of my degree. The interface to the archive shall be based on the ideas in Ka-Ping Yee's paper on his Zest prototype. Over the past two weeks I have been looking at requirements and have the following. These are in no specific order. Due to the time constraints on my project (I am to only spend 200 hours in total on it, including writing reports, presentations etc) there is a limit to the amount I can do. Functional Requirements The archive component should 1. store email discussions. 2. integrate with Mailman. 3. provide a web-based interface to those email-discussions. 4. provide an interface that threads discussions by their content. (ZEST) 5. provide an interface that threads discussions by e-mail replies. 6. allow for full-text searching of the archives. 7. allow for filtering by date, author, and/or topic. 8. be MIME aware. 9. allow archives to be set as public or private. 10. allow posts to be added, deleted, and modified through web interface. 11. allow archives to be locked to prevent modification. 12. allow postings to be emailed. 13. allow postings to be referenced externally. Non-Functional Requirements 1. Maintainable 2. Secure 3. Scalable The minimum I am planning on doing is the first 5 functional requirements restricted by the first 2 non-functional requirements. There are a two reasons I am posting this. Is there anything obvious that I have missed? Which of the functional requirements, 6 to 13, do you feel are the most important? (As part of my report I have to analyse the requirements captured) Any feedback is very much appreciated. Thanks in advance. Iain

14 52

Re: [Mailman-Users] unsubscribe_policy problem? (fixed, includes a patch)
by Pasi Sjoholm Nov. 3, 2003

Nov. 3, 2003

Hello again, anyone didn't reply to me so I fixed this by myself, here is the patch: --- Mailman/Cgi/options.py~ Thu Oct 30 12:02:41 2003 +++ Mailman/Cgi/options.py Thu Oct 30 11:44:55 2003 @@ -156,9 +156,22 @@ def main(): if cgidata.has_key('login-unsub'): # Because they can't supply a password for unsubscribing, we'll need # to do the confirmation dance. + if mlist.isMember(user): - mlist.ConfirmUnsubscription(user, userlang) - doc.addError(_('The confirmation email has been sent.'), tag='') + # If we're doing admin-approved unsubs, don't worry about the password + if mlist.unsubscribe_policy: + try: + mlist.Lock() + try: + mlist.DeleteMember(user, 'via the listinfo page', userack=1) + except Errors.MMNeedApproval: + doc.addError(_('Your unsubscription request has been forwarded to the list administrator for approval.'), tag='') + mlist.Save() + finally: + mlist.Unlock() + else: + mlist.ConfirmUnsubscription(user, userlang) + doc.addError(_('The confirmation email has been sent.'), tag='') else: # Not a member if mlist.private_roster == 0: On Thu, 30 Oct 2003, Pasi Sjoholm wrote: > This is possible when user is not logged in but when logged there will be > a request to unsubscribe user "x" for list admin. So.. it's a bug =) I > also tested it on 2.2.3. > > -- > Pasi Sjöholm > > > On Thu, 30 Oct 2003, Pasi Sjoholm wrote: > > > Hello, > > > > I set up a mailman today and now I have a little problem. I have this > > corporate mailing list and I have set unsubscribe_policy to yes for that > > list but still users can unsubscribe via web gui after they have clicked > > the unsubscribe and confirmed that they really want to unsubscribe via > > email. > > > > I'm using Mailman 2.1.2... is this a bug or what? > > > > -- > > Pasi Sjöholm

2 1

URL change for Reply-To Munging Considered Useful
by Simon Hill Oct. 31, 2003

Oct. 31, 2003

Hello, Please update the URL for my essay "Reply-To Munging Considered Useful" in the help item for setting reply-to. The new URL is: http://www.metasystema.net/essays/reply-to.mhtml Note that the new URL is at metasystema.net rather than metasystema.org. There is no hurry, as metasystema.org will redirect to metasystema.net for another year. However, after that metasystema.org will be retired, so the sooner the change is made, the less users will get a dead link. I'd like to take the opportunity to thank the developers of Mailman for their efforts. I admin several lists and Mailman is my mailing list manager of choice. I'd also like to thank you all for including a link to my essay in Mailman - and an especial thanks to those of you who disagree with my position but decided to include it anyway. Best regards, Simon Hill

1 0

Re: [Mailman-Developers] Re: being flexible.
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Fri, 31 Oct 2003 23:26:08 +0100 Simone Piunno <pioppo(a)ferrara.linux.it> wrote: > On Friday 31 October 2003 22:30, Brad Knowles wrote: >>> Right now we're receiving messages over a pipe and as far as I see >>> we're not using any environment or command line parameter (beside >>> the script name) so where's this SMTP magic? >> You mean the envelope sender address? The IP address of the envelope >> sender? > yes, I mean exactly this, we're not retrieving them, neither by > command line options nor by environment variables. Ahh, yes. I'd forgotten that as I wrap Mailman's wrapper in a larger filter which does take and processes the envelope information. However, there remains a distinct difference if several addresses are conflated to the same POP/IMAP box. In that case you lose the target side of the envelope, and thus can't determine reliably which Mailman-specific address/wrapper/alias to deliver the message to. > Doing it would probably be MTA-specific (i.e. less portable). As of last I checked the Debian Exim packages ship with the no-alias-file-required Mailman configs by default. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: being flexible.
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Fri, 31 Oct 2003 10:21:20 +0100 Simone Piunno <pioppo(a)ferrara.linux.it> wrote: > Alle 01:51, venerdì 31 ottobre 2003, J C Lawrence ha scritto: >>> A test installation (or a poor man's installation) will fetch >>> messages from pop3 mailboxes... >> Hell no. Mailman is a conformant well behaved and very standard mail >> system, not a hack on top of a kludge that deliberately flouts the >> standards just because it wants to. > Hehe, I knew you'd have screamed :) Aye, its a lot of fragile and unnecessary work that encourages a use of mailman which I'd rather never occurred. > Try to think it the other way: a test installation with pop3 is an > hack on top of a solid conformant well behaved and very standard mail > system. Right, but the information lost across the non-SMTP translation is irretrievable and necessary. > Skilled people can do it right now with Mailman 2.1: just configure > fetchmail to act as an MTA and pipe messages to Mailman. The only > difference I proposed is that non-skilled people should be able to do > this... I account this one of those areas where the people who know how to do it, also have some hope of knowing better than to. >> Why? Even ignoring the abuse possibilities, what possible reason >> could we have for that time and effort investment when those problems >> are already far more competently and easily handled than we ever >> could, and there are so many other, more rewarding and demanding >> problems and features on the burner? > We all know CGI is sub-optimal. We're also planning to stop vending > archives directly from disk, increasing the CGI load. True, however the processing delta can be kept quite small. > mod_python or a web runner would perform better. The problem there is that they are insufficiently portable. > I feel the reverse-proxy configuration (similar to zope) would be the > better choice. Why would it be better? What are the disadvantages of those benefits? What specifically useful or necessary function would it add which is not being served today? What interesting and valuable deployment cases would it prevent or make more difficult? Which would it open to the product that are not current addressed? Let's get a cost and value statement going. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 3

Re: Efficient final message disposition (was Re: [Mailman-Developers] Requirements for a new archiver)
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Fri, 31 Oct 2003 16:04:43 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 8:20 PM -0500 2003/10/30, J C Lawrence wrote: >> While I don't disagree, this is really an MTA's job, not Mailman's. >> This is why I've been doing log analysis of MXes and routing mail to >> customised outbound MTAs on the basis of responsiveness, since early >> 2000. Adaptive MX routing is great stuff. > There is a need for this function, and no MTA available today does it. > MLMs throughout the history of the Internet have incorporated a > variety of features for SMTP performance enhancement that are unique > to mailing lists or are usually found primarily in mailing lists, and > this is no different. True. Its not a very difficult process, and is absurdly expensive the way I handle it. At some point in my copious spare time I should whack another couple config tokens into Exim, just to up the ante. > If you want to externalize all these functions outside of mailman, > that's fine. But then someone has to pick up the ball and start > hacking on bulk_mailer or some other program to provide these > features. Aye, but some care should be taken here defining who the people are, between the Good-For-Mailman, and Good-For-Large-Mail-Systems camps. They're related, but not synonymous. >> Yup. I did it at the first level with an initial SMTP proxy which >> routed based on MX response records pulled from a DB. > Again, this is a feature which is not found on any MTA available > today, and which is known to have a huge impact on mailing list > performance. This feature needs to be provided somewhere, by someone. True. > If you go back to Barry's message, he was talking about getting > even further involved, by doing a mail-merge process. Since there is > no MMTP (something that Bryan Costales, Eric Allman, and I had worked > on for a while, before we realized that it would just make the spam > problem worse and then dropped all further efforts), there is a need > for an intermediate program that is called by mailman and then hands > the messages off to the MTA. <nod> Mailmerge and VERP customisation, and the standards for the communication of those things to the MTA are areas that need attention, both for Mailman and the rest of the market (tho the IronPort and related guys might argue). This would be a good point to get some cross-MTA discussion going on. >> We're not in the game of second guessing the MTAs. That way lies >> wasted time and madness. > If there were MLTAs which were optimized for this function, IIRC QMail has a (typically DJB) VERP/rewrite handoff method. I also recall that it is very bound into QMail's process and IO model, but perhaps this should be examined? > I would agree with you. Since we're trying to take standard MTAs > which may have only some optimizations that might be generally > applicable to most situations (including mailing lists), I must > disagree. There's that audience problem again. I actually agree with you in the general case, and am willing to spend time and effort in that direction. However I see this as somewhat disjoint from Mailman in specific. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: Efficient final message disposition (was Re: [Mailman-Developers] Requirements for a new archiver)
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Thu, 30 Oct 2003 18:20:56 +0100 Brad Knowles <brad.knowles(a)skynet.be> wrote: > At 8:41 AM -0800 2003/10/30, Chuq Von Rospach wrote: > One of them is recipient sorting by average delivery time over the > past week (probably want a decaying geometric mean), which would > require tracking log data on a per-recipient basis. While I don't disagree, this is really an MTA's job, not Mailman's. This is why I've been doing log analysis of MXes and routing mail to customised outbound MTAs on the basis of responsiveness, since early 2000. Adaptive MX routing is great stuff. > Another is two-level message handling, by configuring the MTA for the > initial delivery attempt to use very low timeouts, but then to fall > back to a secondary MTA (or MTA pool) that uses more standard timeouts > for those sites that are slower. Yup. I did it at the first level with an initial SMTP proxy which routed based on MX response records pulled from a DB. > Perhaps in its current form, that is true. However, not all sites are > using sendmail 8.12, and of the ones that are, most are probably not > using it in a manner that is more suitable for mailing lists. I'm generally of the view that Mailman should do opportunistic domain sorting and per-MTA customised VERP handoffs (because nobody has standardised VERP across MTAs), and beyond that to back off. Mailman's job is to get the outbound mail into the MTA's spool as quickly as possible, wrapped in transactions (ie RCPT TO bundles) that are friendly to efficient processing, and that's it. We're not in the game of second guessing the MTAs. That way lies wasted time and madness. > However, given the issues you've mentioned, it would probably be a > good idea to be able to turn off selected "bulk_mailer" type features, > so that you can let the MTA do more of it's job better -- if it is > configured to do so. There are thresholds for covering up for broken software. There are also thresholds for covering up for SysAdm negligence or oversight. You've got to pick where you stop accepting the problem. Ideally we should be resilient and friendly to both. Realistically we need to do something reasonable and not worry too hard about the rest. Priorities. Mailman's primary performance problems are not at the MTA hand off. MTA configuration and tuning for mailing lists is only a minor art. There is not-inconsiderable documentation and understanding of the field. A US$2K commodity box subjected to moderate tuning efforts using readily available documentation can sustain 2,400 outbound deliveries per minute. You do the arithmetic. In a perfect world that maps out to 3.4 million per day. Cut that under half for queue injection overhead other crap and you're still talking a million deliveries per day for a US$2K host.[1] A million messages a day already puts us above the 99th percentile for list server audiences. I'm not really concerned about that problem. Where Mailman's performance hurts is in the handling of the list configs, especially for lists with very large memberships rosters and in queue runner performance and overhead (try watching queue runner's system resource profile in v2.1 for lists with > 50,000 members). For me those are the obvious low hanging fruit, and those are the points that will help not just the performance hounds, but also the lower 80% who are running under-provisioned under-configured under-admined multi-purpose boxes who want Mailman to be a bit more reasonable and forgiving about their not-so-brilliant systems. [1] That's of course assuming reasonable sustained queue size and responsive MXes. However, those are separate problems and ignoring MTA-specific behaviours (like Exim's active hatred of large queues), the methods and systems to segment and tame those problems are fairly well known. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: being flexible.
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Fri, 31 Oct 2003 00:45:32 +0100 Simone Piunno <pioppo(a)ferrara.linux.it> wrote: > On Thursday 30 October 2003 05:17, J C Lawrence wrote: >>> ...as well as implement a bulk mailer to eliminate the need for an >>> outgoing mail server. >> Eeeek! I trust this would be for immediate handoff to a "real" MTA >> versus handling final delivery directly? Quite the Pandora's box if >> not. > I believe the best approach is to cover all options: No. This is an one-size-doesn't-fit-anybody argument. > A test installation (or a poor man's installation) will fetch messages > from pop3 mailboxes... Hell no. Mailman is a conformant well behaved and very standard mail system, not a hack on top of a kludge that deliberately flouts the standards just because it wants to. > ... (polling! I can hear you scream while you read this!) and send > them directly to the internet (no real MTA involved) and will probably > serve web pages directly, controlling port 80 (no real web server > involved). Why? Even ignoring the abuse possibilities, what possible reason could we have for that time and effort investment when those problems are already far more competently and easily handled than we ever could, and there are so many other, more rewarding and demanding problems and features on the burner? -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: Efficient final message disposition (was Re: [Mailman-Developers] Requirements for a new archiver)
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Thu, 30 Oct 2003 09:53:19 -0500 Barry Warsaw <barry(a)python.org> wrote: > - The desire/requirement that Mailman chunk and sort recipients This shouldn't be any more complex than domain sorting, and need not be perfect. > - The ability for Mailman to swamp the mail server or cause the mail > server to consume all available cpu Rate limiting. > - The fact that failures in upstream mail server are reported to > Mailman as bounces instead of as error codes I don't know that Mailman can do anything about this. We can't reliably distinguish between system errors and delivery failures for MTAs beyond Mailman's borders. There's a protocol hole here I don't know we can or should attempt to fix. > - Inefficiencies in VERP/personalization/mail-merge because of the > lack of cooperation Oh yeah. > - The need for Mailman to queue outgoing messages that aren't > completely delivered Queue runner could do with some more intelligence in that dept. > Mailman wants to simply hand that data off to some agent and forget > about it. It wants to know that the agent will make best effort to > mail merge and deliver. It wants to be informed of any final delivery > failures. And that's it. Mailman doesn't want to chunkify > recipients, and it doesn't want to sort them. It doesn't want to > worry about a mail server effectively managing system resources. I'd > rather not have to hand it a couple of meg of recipient or > substitution data, but there seems to be no other way. > So what can we do here to improve matters? Start yelling at DJB, Wietse, Phillip, and Eric about a standardised SMTP extension for VERP. With a little luck and minor work we can probably get some of the other commercial mail people involved as well. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: Efficient final message disposition (was Re: [Mailman-Developers] Requirements for a new archiver)
by J C Lawrence Oct. 31, 2003

Oct. 31, 2003

On Thu, 30 Oct 2003 08:41:17 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Oct 30, 2003, at 7:48 AM, Brad Knowles wrote: > One big reason: increasing spam blocking (stupid or otherwise) of > non-individually addressed email. The old list server setup of: > to: subscribers of list <list@foo> > bcc: bulk_drop(a)of.subscribes > is increasingly risky as far as delivery is concerned. I've seen a couple mail BCPs and internal spam-handling plans at large ISPs and corporates which explicitly include the line item: Discard all mail with more than one address in the envelope. Scary, stupid, true: They want the pain to stop. I find it hard to blame them. > I also don't think it allows for the kind of personalization that's > needed for your general audiences (help URLs, unsub URls, etc). Aye, such VERPish attributes is becoming a necessity. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0