Mailman 3 June 2001 - Mailman-Developers

search engine indexing is crucial to success!
by admin＠xpertindexing.com June 19, 2001

June 19, 2001

Search engines can account for up to 80% of website traffic. Having a beautiful site is important but without the traffic there is no one to appreciate it and consequently NO BUSINESS. Indexing with the search engines seems as simple as filling in the blanks but getting good positions is NOT as easy! Our staff can research your site and get you serious shoppers by getting your site up in the top 15 positions with keyword strings that work. Please visit our site to see how we can help you get the exposure you need to be successful. http://www.xpertindexing.com Thank you for your time and attention Kind Regards, Graeme Skinner - Administration admin(a)xpertindexing.com To be removed from our email list please click here http://www.xpertindexing.com/X/remove.html?email=mailman-developers@python.…

3 2

Handling non-fatal "bounces"
by Carson Gaspar June 19, 2001

June 19, 2001

So we converted the firewalls mailing list to mailman, and we're now being inundated by bounce messages that mailman isn't handling. I'm going to write bounce modules for the actual bounces, but I can't figure out how to handle non-fatal errors (message could not be delivered, will retry soon, etc.). What I want to do is ignore the bounce (don't flag the list member), but _not_ forward the message to the list admins. The current API seems incapable of doing this. Am I missing something? -- Carson Gaspar - carson(a)taltos.org Queen trapped in a butch body

5 4

Re: [Mailman-Developers] Handling non-fatal "bounces"
by J C Lawrence June 18, 2001

June 18, 2001

On Mon, 18 Jun 2001 17:42:31 -0400 Jay R Ashworth <jra(a)baylink.com> wrote: > On Mon, Jun 18, 2001 at 05:21:19PM -0400, Barry A. Warsaw wrote: >> One question: should Mailman support doing an (optional) >> auto-kvetch back to the site's postmaster? > Two words: "Sorcerer's Apprentice". Limit one per day. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

1 0

Re: [Mailman-Developers] Re: To VERP or not to VERP?
by J C Lawrence June 18, 2001

June 18, 2001

On Sun, 17 Jun 2001 12:57:20 -0400 Jay R Ashworth <jra(a)baylink.com> wrote: > On Sun, Jun 17, 2001 at 09:53:28AM -0700, Chuq Von Rospach wrote: >> But what I've found, for really large e-mail installations, >> there's always another bottleneck. The bigger/faster machine >> paradigm just doesn't scale after a while, so what I'm working on >> now is a new setup that I'm calling the "army of smurfs" >> design. I'm going to be buying lots of small/fast/cheap boxes, >> and not going to try to try to keep making that single monolithic >> machine do incrementally more. > If you're talking about generic large mail farms, Chuq, you > *really* need to go find the Earthlink white paper on that and > read it, if you haven't already. They have one on news, too. > Don't recall the URL; Ask The Web<tm>. http://www.earthlink.net/about/papers/mailarch.html http://www.earthlink.net/about/papers/scalenews.html -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

2 1

Re: [Mailman-Developers] Re: To VERP or not to VERP?
by J C Lawrence June 18, 2001

June 18, 2001

On Sun, 17 Jun 2001 20:58:58 +0200 fil <fil(a)rezo.net> wrote: > On my setup (postfix) there's a heavy use of the > /etc/postfix/transport feature, that redirects all domains with > which I have a problem (like those that are not always accepting > mail I send) to a slave machine that is very near (in terms of > bandwidth) the first one. Back when I used to do this under Exim by parsing the mainlog for messages which arrived long before their delivery time. Given a frequent high enough score on this, I'd then do a domain route to a similar Bad Mail box. It was nice to automate it. No attention required. > It reduces A LOT the disk problems (mostly mailqueue becoming very > heavy) -- maybe I should continue this way with more domains being > handled to slave machine(s). Did you have any automated way to collect and maintain the slow MX list? One thing I never did was to implement a system to rescind the "slow" verdict on MXes should they come back to life for a reasonable period. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

2 1

Re: [Mailman-Developers] Re: [Mailman-Users] Allowing users to join without specifying pas swords
by J C Lawrence June 18, 2001

June 18, 2001

On Sun, 17 Jun 2001 00:46:47 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Friday, June 15, 2001, at 01:19 PM, Barry A. Warsaw wrote: > I would set Mailman's 2.1 default to have this turned ON... Agreed. > Barry's mileage may vary on his preferences for default, of > course, and it's his show. I think the advantages of the > customized URL/email capability is a huge one and most sites will > benefit from it -- but the network hit might kill some sites, so > we have to give them an easy ability to turn the feature off. You also ignore local physical resource consumption. Network bandwidth and expense are not always the only constraining factors. Mail systems tend to be disk IO constrained in addition to network bound with file creat()/open() being the biggies. Given a busy box that tends to maintain a significant backlog, under VERP/RCPT_TO=1 inode exhaustion becomes a more probable case than with larger envelopes (similar/identical curves to above). Yes, real mail admins know to mkfs /var/spool with a higher inode count than the rest of their systems. I'll let you guess what percentage of mail admins actually do. There are quite a few lists out there running off i486 boxes with small disks. Some of them have nice bandwidth but have horrible disk IO. ObExample: Up until fairly recently svlug.org (which runs mailman) was an i486. Excellent outbound bandwidth. Lousy local physical resources. Subscriber base there is in the low/middle single digit thousands IIRC. (Thanks to VA things are much better now) The README should enclude notes on these factors as well. > What do y'all think? I suspect Nigel Metherington is going to chime in here, I'm interested in also seeing the percentage case, if only for better bounce handling. > I've included mailman-developers on this reply, since while this > started on mm-users, it really ought to be discussed on the > developers list... Damned right. > Yes, I am an agent of Satan, but my duties are largely ceremonial. How topical. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

2 1

Re: [Mailman-Developers] Re: To VERP or not to VERP?
by J C Lawrence June 18, 2001

June 18, 2001

On Sun, 17 Jun 2001 14:23:32 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > This isn't a discussion-list system, but the delivery setup could > be adapted to that fairly easily. If you were going to do > something similar on Mailman, you could do it by having a farm of > SMTP outgoing machines under a round-robin with a short time-out, > and making sure qrunner re-looks up the IP after every message. Do > that, and then extend to allow for parallel qrunners, and you can > buidl a heck of a mail list farm. 2.1 will do most of that fairly > easily, the rest is figuring out and configuring the SMTP smurf > farm and its round-robin. Its arguably cheaper/easier to instead us a load balancer box as the SMTP target which then does a per-transaction round-robin to the target nodes in the farm. Fairly easy to do, just a little proxy, and then let rip. Mailman never knows any different, you've got management abstraction in the proxy box, and can add, remove, and fail out nodes transparently without any cacheing problems torquing the queues. > The downside is you add administrative complexity, plus you need > to engineer getting the data where it needs to be (and the > security of making sure nobody else uses your smurfs...). Sit the smurf farm behind a router that redirects all inbound port 25 connections to <where-ever>. At that point the only access to the smurfs is from the inside via your proxy box as above. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

1 0

Re: [Mailman-Developers] Re: To VERP or not to VERP?
by J C Lawrence June 18, 2001

June 18, 2001

On Sun, 17 Jun 2001 09:53:28 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Sunday, June 17, 2001, at 09:39 AM, J C Lawrence wrote: >> Exim for instance will defer deliveries if more than N messages >> are received in a single connection. As a result, you typically >> get no outbound deliveries going on during a qrunner broadcast. > IMHO, I consider that seriously broken. It can also be viewed as an advantage. Given a large inbound mail load, Exim throttles itself to try and be maximally available/responsive for any other large mail events that might happen synchronously. This is a seemingly fairly standard philosophy of Exim's: Make sure you can always play nice. > Funny, I don't remember being absent minded. Neither did I. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

1 0

Re: [Mailman-Developers] Re: To VERP or not to VERP?
by J C Lawrence June 17, 2001

June 17, 2001

On Sun, 17 Jun 2001 08:41:56 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > the amount of disk it takes isn't an issue (within reason) -- > remember, it's going to start sending right away, so message will > be gone out of the queue. This is not a safe assumption. Exim for instance will defer deliveries if more than N messages are received in a single connection. As a result, you typically get no outbound deliveries going on during a qrunner broadcast. > You don't queue it all up and then start delivery. Uhh. > But this generates disk IO, and that can start bogging you down if > you don't tune things properly. That means taking advantage of > sub-folders in your mail queue for any MTA that allows them (the > #1 performance death for a typical sendmail system: write-locks on > /var/spool/mqueue, since every sendmail process has to create > files in that directory. If you haven't set up subfolders, you are > (I say this in a nice way) an idiot. if you aren't using a version > of sendmail that allows for them, I'll call you an idiot in a > not-nice way. Anyone not running at least 8.10 is hosed, so forget > them... Great READ/FAQ topic. > Most MTAs are sensitive to this and work to minimize the > impact. Even sendmail finally figured it out. But you still need, > in large e-mail environments, to look at splitting this across > heads and spindles. My experiments have indicated you're better > off having mail on separate spindles than you are building a RAID > using those same spindles, for whatever that's worth. I've done similar experiments with news spools. The values are incredibly subjective to RAID stripe size. Bob Mende IIRC did some interesting work here which I think he presented at LISA. FWIW We ended up with a 1Meg strip size as the sweet spot, much bigger than we'd expected. > And if you have lots of RAM, you can start using ram disks, and > then you have lots of fun... (yes, I've done that. It's amazing > how much faster sendmail is when you remove the disk I/O on those > directory inodes...) Silicon disks are another one. At Critical Path we used solid state disks for /var/spool and Qmail fairly, umm, whirred. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

5 5

Re: [Mailman-Developers] Re: To VERP or not to VERP?
by J C Lawrence June 17, 2001

June 17, 2001

On Sun, 17 Jun 2001 09:53:28 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Sunday, June 17, 2001, at 09:39 AM, J C Lawrence wrote: >> Exim for instance will defer deliveries if more than N messages >> are received in a single connection. As a result, you typically >> get no outbound deliveries going on during a qrunner broadcast. > IMHO, I consider that seriously broken. Its configurable. >> I've done similar experiments with news spools. The values are >> incredibly subjective to RAID stripe size. Bob Mende IIRC did >> some interesting work here which I think he presented at LISA. >> >> FWIW We ended up with a 1Meg strip size as the sweet spot, much >> bigger than we'd expected. > I know someone who did the same, and found the best perforamnce > iwth (I kid you not) 32 meg strip sizes. Not surprised. Among other things its going to be sensitive to number of spindles, RAID type, and feed characteristics. >> Silicon disks are another one. At Critical Path we used solid >> state disks for /var/spool and Qmail fairly, umm, whirred. > But what I've found, for really large e-mail installations, > there's always another bottleneck. The bigger/faster machine > paradigm just doesn't scale after a while, so what I'm working on > now is a new setup that I'm calling the "army of smurfs" > design. I'm going to be buying lots of small/fast/cheap boxes, and > not going to try to try to keep making that single monolithic > machine do incrementally more. I've noticed that E-Machines are getting popular for precisely that use/need. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

1 0