Mailman 3 July 2001 - Mailman-Developers

mailmanctl patch
by John Read 25 Jul '01

25 Jul '01

Hi, after installing the new beta version I found that not having starting, and stopping messages in the mailmanctl was confusing when it was included in the Linux startup scripts. All other system services have a short message stating what they doing during the boot phase. I have included the start, restart, and stop messages into mailmanctl, and included the changes in a short patch file. (Its only three lines of code so dont get excited!!) Otherwise the installation, and testing so far has been without any problems. Great software. -- John Read NewNet Marketing Tel: +49-8076-8879818 Fax: +49-8076-8879819

2 1

Re: [Mailman-Developers] Some notes on my stuff...
by J C Lawrence 24 Jul '01

24 Jul '01

On Mon, 23 Jul 2001 19:56:15 -0700 Graham TerMarsch <graham(a)howlingfrog.com> wrote: > I know Sendmail is generally chastized as being slow and doggish > as far as SMTP servers go, but from having twiddled with it enough > lately, I'm beginning to wonder if thats just because the default > config that it builds with is sluggish (as I've been able to get > ~8-10x throughput from the same server just from reshuffling and > retuning the config and the queue runners). Older versions of sendmail had a very deserved reputation for being asleep at the wheel in the performance arena. Recent versions of sendmail have considerably improved, especially in their queue handling algorithms. They're still not up to the level of the competition (specifically exim, postfix, qmail), but they've taken at least an order of magnitude out of the gap. How I look at it: Pro: Many old timers know sendmail and are comfortable with Sendmail. Sendmail is the default MTA on most (all?) commercial *ix. There are readily available monitoring tools for Sendmail (snmp etc) Supports UUCP better than most. Con: Unreadable/impractical/painful config file -- exim/postfix/qmail/smail all have very human readable and well documented configu files. Mediocre if reasonable performance. Requires considerable work to tune reasonably. Cannot turn off DNS verification for messages received on without also universally disabling SPAM checks (per Chuq). Unimpressive security history. SMail is a lot slicker and easier for UUCP mail than Sendmail Poor spool administrative tools (cf eximon) While endlessly configurable/extensible, such extensions are sufficiently opaque as to be effectively write-only (cf exim and postfix in particular) -- J C Lawrence )\._.,--....,'``. ---------(*) /, _.. \ _\ ;`._ ,. claw(a)kanga.nu `._.-(,_..'--(,_..'`-.;.' http://www.kanga.nu/~claw/ Oh Freddled Gruntbuggly

1 0

Some notes on my stuff...
by Chuq Von Rospach 24 Jul '01

24 Jul '01

We had discussions a while back on the performance problems I was seeing on my big mailman machine. I figured it was probably time to update, so everyone knew what was going on... For those that don't remember, my big mailman machine (Sun E250, mailman 2.0.5, sendmail) was maxing out at about 625 messages processed a day -- and we were seeing peak backlogs of 4 hours between receiving and processing. Not good. My testing showed the major processing delay was the speed of uptake by sendmail, primarily due to DNS delays. And given that you can't turn off DNS lookups without turning off a lot of the anti-spam stuff, you're kinda stuck. So I've been working to move my system sfrom sendmail to postfix. Along the way, by pure happenstance (you can read that to mean "I was testing, I screwed up, and it showed me something I wasn't looking for and didn't expect to find"), I found out that there was a second, more subtle and infernal problem that was making the main problem even worse: Disk I/O It turns out I was saturating a disk spindle, and it wasn't obvious that I was doing it. Once I found it, though, it was obvious what was happening -- sendmail was, basically, stuttering, and with the disk I/O causing delays, every minor delay by DNS was being elongated, which was causing much of the performance problem. I was able to fix most of the disk I/O problem by splitting off a big chunk to an underused spindle, and saw performance improve markedly -- it's now processing 700+ messages a day (the peak day was 810), and my delay to delivery is normally 5 minutes or less. I've seen a few times where that delay's grown to a whopping half an hour -- but that, at least right now, is fine. Because of this, I've slowed down on the move to postfix, not because I don't plan on doing it, but because other things have taken priority for now and I want to do some testing before I make the change; as I like to tell my management, swapping MTAs is like replacing the transmission in a bus just before driving the kids to summer camp, and I don't particularly want to end up on the side of the road because I was in too much of a hurry...). My capacity magically went up at least 30% by fixing the disk I/O problems. Which leads to a few truisms. First, when you're trying to find a problem, assume nothing. And once you FIND the problem, don't assume it's THE problem, it might instead be A problem -- in my case, there was a second, underlying problem that the switch to postfix would have reduced but not solved, and it's unclear by how much. And, of course, check everything. One thing I didn't look at until after I stumbled into this was spindle usage, and where I assumed things were okay, it turned out -- it wasn't. And it's probably safe to say you should ALWAYS assume that disk I/O is a problem, and rule it out before you start finding other problems... Because it's something that will make all of the other problems worse, but not necessarily easy to find unless you look for it specifically. We're now talking about upgrading all of the email machines to RAID 1+0 down the road, just to build a system that maximizes the disk performance, since it's now clear I've been underestimating it's impact (even though I thought I wasn't...) -- Chuq Von Rospach, Internet Gnome <http://www.chuqui.com> [<chuqui(a)plaidworks.com> = <me(a)chuqui.com> = <chuq(a)apple.com>] Yes, yes, I've finally finished my home page. Lucky you. The first rule of holes: If you are in one, stop digging.

5 4

"message has implicit destination"
by Dan Mick 23 Jul '01

23 Jul '01

So I understand this perfectly, but apparently I'm a freak; admins and users all over seem to misconstrue this. If it hasn't yet been changed for 2.1, I recommend something more like "List address did not appear in To: or Cc: headers, perhaps because the message was sent with the list address in a Bcc: header."

1 0

Wooz.org temporary outage
by barry＠digicool.com 20 Jul '01

20 Jul '01

My DSL ISP (who does a great job, BTW) is switching my domain from its New York City POP to its Washington DC POP in a few hours. I already get great service, so I'm hoping that switching to the local POP will at least not make it any worse; hopefully it'll be better still. Anyway, this means that my IP addresses are all changing, and I'll be updating my DNS information shortly. So wooz.org (the terminus for all my email, including via python.org and digicool.com) will be off-line, hopefully for just a few hours. I've got a secondary MX setup which appears to work, so I shouldn't lose any email. I just wanted to send out a notice in case you guys see any bounces over the next 24 hours or so. Cheers, -Barry

1 0

Another big batch of checkins coming
by barry＠digicool.com 20 Jul '01

20 Jul '01

I have another big batch of changes that are about to be checked in, and I wanted to give CVSers a heads up that there might be a bit of instability as all the commits go through. I promise to get back to the raging GET vs. POST debates after this "branch" is merged. The changes that are coming grew out of some discussions Dan Pierson of Control.com and I had at the 9th Python conference earlier this year. The features were fairly high on Control.com's wish list, and I felt that these features and changes were good for Mailman. Control.com funded some focussed work in these areas and the results of those efforts are what I'm going to be checking in. I want to thank Control.com, and especially Dan and Ken Crater for their support! From my perspective this kind of sponsorship was very successful, and could serve as a model for others willing to fund some Mailman development work <hint> <hint>. :) There are three general features that are being added. A brief overview follows: - a topic filter - membership adaptors - "virtual" mailing lists Topic filters allow a mailing list admininstrator to define topics, based on regular expressions, that list members can use to filter out some portion of a list's traffic. Topic matching is performed on the Subject: and Keywords: headers (with optional extension into message bodies). Individual users can select the topics they are interested in and will only receive message that match those topics. Topic filtering only works for regular delivery; topics have no effect on digests. Membership adaptors will allow a mailing list to get its membership information from any external source. Essentially, it's an API for accessing and changing all information related to a list's subscriber. The API was inspired in part by Chris Ryan's Datasource Implementation Template (see the Mailman wiki), although it only covers membership information (similar to Chris's MemberData class), and the API reflects my own design preferences. There is currently only one implemented adaptor: OldStyleMemberships which use all the traditional data structures in the MailList object. I've played with adaptors that can extract membership information from flat files, and have toyed with a ZODB based membership adaptor, but neither of those are ready for prime time yet, so I won't be checking them in. [Aside: although not sponsored directly by Control.com, the membership adaptors made it easy to add support for Real Names, so that will be part of the check too.] [Aside: additional ancillary changes allow MailList instance customizations via an extension framework. Also, the web GUI parts of the admin interface are being further componentized for easier customization.] Finally, there will be support for "virtual" mailing lists. Essentially, you create a normal list, but there are now two external interfaces which allow you to send a message to an explicit set of recipients, who need not be members of the list. The message to that set of members will look like it's coming from the real mailng list. The two interfaces are 1) a Python module you could import from your application, through which you could call Post.inject() with a message object, a list name, and a list of recipient email address; 2) a command line interface which an external process could call to pass the same information to Post.inject(). Enjoy, -Barry

3 3

list throttling, handling auto responders
by Chuck Moss 19 Jul '01

19 Jul '01

I recall a discussion a while ago about ways to handle autoresponder mail loops but I can't find the thread at the moment. Not sure what the resolution was but I remember some of the ideas/issues. Problem: How can the software automatically handle mail loops caused by broken autoresponders. -------------------------------------------------------------------- Option 1: detect loops by frequency of posts from an individual account. Throttle the list to a certain number of messages/user in a given time frame. Advantages: easy to implement? relatively robust? Disadvantages: can impact lively productive discussions if limit is set too low. Useless if too high. Some mail loops may be too slow to be caught by this. Assume mail corporate mail queue only cycles every 15 minutes or so. Lots of moderator work. -------------------------------------------------------------------- Option 2: detect loops by type of content in messages. try to bounce/moderate bad posts. Advantages: if functional no impact on lively discussions. Disadvantages: complex implementation, difficult to catch every instance -------------------------------------------------------------------- This happened on another list the other day and I had an idea. I am hoping brain storm but maybe brain drizzle. How about using option 1 with a modification. If the user hits a threshold their postings would become self-moderated. A post could result in an email message to the sender similar to the message they get when awaiting moderator approval. It would allow them to apporve their own message either by email or web interface. If the web interface was used they could approve several pending posts at once. The email self-moderation would need to be complex enough that a simply autoreply could not trigger the approval. This "self-moderation" feature could be extended to allow for postings from unsubscribed addresses etc. Sorry for the term self-moderation. It is really just a confirmation similar to subscription. Hopefully the code could be re-used. Has this been done before? What major pitfalls to this approach am I overlooking? Chuck Moss

1 0

Please Explain !!!
by Simon Troup 19 Jul '01

19 Jul '01

Can anyone please explain why, after joining the Mailman Users mailing list, I have been bombarded by dozens of emails that appear to orinate form mailing lists at portal2.com and outblaze.com and are addressed to sardy5(a)graffiti.net (not me !! My email appears in "Envelope to"). I also received over 4,000 (yes 4,000) bounced emails telling me that my (not actually my but sardy5(a)graffiti.net) subscription to some mailing list had failed. This has wasted an entire day for me. If you know what link there is between Mailman and portal2.com or outblaze.com please let me know. How could the emails get mixed up like this? I have now received mail from the Mailman Developers list addressed to sardy5(a)graffiti.net which is why I am posting here. My emails to outblaze.com have gone unanswered and now I am desperate to stop this unwanted bombardment of mail. Is someone collecting email addresses from the Mailman mailing lists ??? Any help greatly appreciated, I only want to receive my own mail, and getting dozens and dozens of misdirected mail on my sad English 56k connection is killing my system. Simon Troup simon.troup(a)digitalmusicart.com

4 8

Re: [Mailman-Developers] Re: GET vs POST (was Re: subscription confirmations)
by J C Lawrence 19 Jul '01

19 Jul '01

On Thu, 19 Jul 2001 00:41:20 -0400 Jay R Ashworth <jra(a)baylink.com> wrote: > My apologies that this is ugly; I'll try to clarify all my > assertions here to avoid wasting bandwidth. Using an MUA/editor which handled quote line wrapping properly would have helped quite a bit. Suggest SuperCite. If you're one of those desperate vim users per recent SVLUG traffic VIM has a mode/macro/tool/something to properly handling quote wrapping. > Perhaps I'm mistaken, and a POST *can* be called with a ? > parameter, but in general, if you see a ?, the call will be the > default GET, and if the page is sending parameters as a POST, then > they'll be sent "out of band" to the URL, and the method will be > POST instead of GET. Nope. Do a packet sniff and watch. Its still a GET, its just that most systems will read variables off the URL as if they had been POSTed -- J C Lawrence )\._.,--....,'``. ---------(*) /, _.. \ _\ ;`._ ,. claw(a)kanga.nu `._.-(,_..'--(,_..'`-.;.' http://www.kanga.nu/~claw/ Oh Freddled Gruntbuggly

2 1

Re: [Mailman-Developers] Re: GET vs POST (was Re: subscription confirmations)
by J C Lawrence 19 Jul '01

19 Jul '01

On Thu, 19 Jul 2001 00:29:22 -0400 Jay R Ashworth <jra(a)baylink.com> wrote: > A URL with a "?" character in it is known to be dynamic -- that > is, it's *expected* not to return the same content, depending on > what parameter it's given; this is why search engines customarily > ignore them... and it what I meant by "magic", in the context in > which I was using it. This presumed fact would seem a myth. Kanga.Nu has circa 150K web pages of which between one third and one half have URL variables ('?' in the URL). __ALL__ of those pages are regularly (evey month) and repeatedly indexed by Google, Inktomi etc. Actually at this point I'm not aware of any english language search engine which doesn't index Kanga.Nu (encluding those pages), and I know its also indexed by several foreign language search engines as well. Simple example: http://www.google.com/search?q=beej+site%3Awww.kanga.nu First hit: http://www.kanga.nu/library/index.php3?viewCat=10 A dynamically generated page whose content depends on the value of "viewCat". Aside: The Library At Kanga.Nu (of which that is a page) is largely offline currently due to DB corruption (MySQL has been eating its own databases and I've not had a chance to restore from backups). Please don't email me asking why or telling me that all the root categories are missing. Yeah, I know. -- J C Lawrence )\._.,--....,'``. ---------(*) /, _.. \ _\ ;`._ ,. claw(a)kanga.nu `._.-(,_..'--(,_..'`-.;.' http://www.kanga.nu/~claw/ Oh Freddled Gruntbuggly

4 4