Mailman 3 June 2001 - Mailman-Developers

Re: [Mailman-Developers] Re: [Mailman-Users] Allowing users to join without specifying pas swords
by J C Lawrence 17 Jun '01

17 Jun '01

On Sun, 17 Jun 2001 08:29:50 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > On Sunday, June 17, 2001, at 06:21 AM, alex wetmore wrote: >> I would argue that it should default to OFF > I'm not surprised. It's Barry's call, but I think the customized > URL is useful enough we want people to use it unless they have to > turn it off, we don't want to have to try to convince the people > who install stuff and leave everything defaulted to turn it on. I spent some time on both sides of the fence as to which way to default it. I came out out on the ON side. Reasons: There's a significant feature gain. Most of those new features help list members. All of them help the list owners. With VERP they also help the system over the longer term. Mailman becomes even more user friendly (by default). The default to OFF case has the biggie: For most people, nothing will break and they'll have no surprises. This is caveated by the fact that the only people that things will break for are those that didn't read the README. -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

1 0

Re: [Mailman-Users] Allowing users to join without specifying pas swords
by Chuq Von Rospach 17 Jun '01

17 Jun '01

On Friday, June 15, 2001, at 01:19 PM, Barry A. Warsaw wrote: > CVR> points. but we need to quantify what those points are and > CVR> what the impact is, so we can decide just how to move forward > CVR> on this. > I'd love to see any statistic you (or anybody) gathers on this > subject. It's definitely intriguing, but right now I don't have the > time or systems to do this kind of data gathering. > Okay, here's a first cut at some data. I'm going to assume the following: 1000 subscribers -- no digest subscribers to simplify this. Assume just individual messages. The message size is 10K, including header. The bandwidth needed to generate a connection to send a message is 1K (which is pretty close) The bandwidth needed to add an address to an existing message is about 1/10 of a K (also pretty close). The practical limit to the number of messages you can piggyback is 100, since this is specified in RFC2821 as the smallest number a site is REQUIRED to take. In practice, due to non-conformant sites, you have to be careful setting it beyond 50 these days, because sites set this number down because they think it slows down the spammers (I'm yet to be convinced it makes a damn bit a difference, especially since MTAs like postifx recognize the 452 and auto-adjust now. This is another place where sendmail seems behind the technology curve, FWIW) How much bandwidth is used depends on these factors: what your piggyback value is (in mailman, it's SMTP_MAX_RCPTS) how many domains have > 1 subscriber. Here's how plaidworks breaks down: 3101 subscribers across 1287 domains. that's an average of 2.3 subscribers per domain, but the numbers skew wildly, so averages are meaningless. Here's how my site breaks down: # of subscribers # of domains/# of users --------------------- ----------------- 1 263/263 2 142/284 3 40/120 4 19/76 5 16/80 6 10/60 7 7/49 8 3/24 9 6/54 10 2/20 11 2/22 12 2/24 13 1/ 14 1/ 16 1/ 17 1/ (worldnet.att.net) 22 1/(juno.com) 29 1 (mindspring.com) 30 1 (pacbell.net) 35 1 (plaidworks.com) 43 1 (sympatico.ca) 53 1 (earthlink.net) 150 1 (home.com) 173 1 (yahoo.com) 228 1 (hotmail.com) 441 1 (aol.com) if you're scoring at home, 37% of subscribers come from that last 4 domains: 5% for home and yahoo, 7% for hotmail, and 14% for aol. those are your 500 pound gorillas (AOL is 800 pounds), and piss them off at your own risk. At the other end, 8% of your users are the only subscriber from a domain. 16% are 1 or 2 per domain. 26% are on sites with 5 or fewer subscribers. Time for some numbers. Back to the 1000 member list for simplicity. The subscriber list breaks down to: 85 - 1/85 45 - 2/90 12 - 3/36 6 - 4/24 [...] 48 - 1 55 - 1 73 - 1 142 - 1 That's 553, or 55% of the subscribers, wedged tightly on both ends of the curve. We can extrapolate what they'll do to bandwidth from the end cases if we need to. Extreme case: SMTP_MAX_RCPTS = 1. 1000 subscribers * (10K message size + 1K overhead) = 11,000K bytes bandwidth. Extreme case: SMTP_MAX_RCPTS = 100 These get sent down the line this way: 85 * 11K 45 * (1 * 11K + 1 * .1K) 12 * (1 * 11K + 2 * .1K 6 * (1 * 11K + 3 * .1K) [...] 1 * 11K + 47 * .1K 1 * 11K + 54 * .1K 1 * 11K + 72 * .1K 2 * 11K + 140 * .1K Do you see how I got these numbers? In the case of the 12 domains with three subscribers, you have to make an 11K connection for the first message, and piggy back on the other two addresses at .01K each. You don't really see huge savings until the big domains, and you'll see AOL goes over the 100 address limit so gets split into two different messages. For this 55%, the SMTP=1 is 6050K. For 100, it's 1711K bytes. That's 28% of the first number, so we're cutting 72% of the bandwidth by chunking at 100. The tradeoff is performance, though -- it takes a lot longer to deliver those AOL addresses, because if you split it into two batches, you can't parallelize the delivery. Package up 100 AOL addresses in one batch, none of them get delivered until all 100 addresses are sent to AOL and accepted. It's much faster to send them as ten batches of ten in parallel -- but that's the trade off here. Cut network bandwidth but slow delivery to the larger domains. Okay, let's look at a case in the middle. SMTP_MAX = 5. The ones with less than 5 don't change, but the big domains do 85 * 11K 45 * (1 * 11K + 1 * .1K) 12 * (1 * 11K + 2 * .1K 6 * (1 * 11K + 3 * .1K) [...] 1 * (10 * 11K + 38 * .1k) 1 * (11 * 11K + 44 * .1K) 1 * (15 * 11K + 58 * .1K) 1* (29 * 11K + 113 * .1K) that works out to (trust me) about 2378K, or about a 60% reduction. Let's try SMTP_MAX = 2. 85 * 11K 45 * (1 * 11K + 1 * .1K) 12 * (2 * 11K + 1 * .1K 6 * (2 * 11K + 2 * .1K) [...] 1 * (10 * 11K + 38 * .1k) 1 * (11 * 11K + 44 * .1K) 1 * (15 * 11K + 58 * .1K) 1* (29 * 11K + 113 * .1K) that works out to 2575K, or about a 57% cut. By a rough look at those domains in the middle, I'd say these numbers are good +-10%. What's this mean? Here's the executive summary: The network penalty between SMTP_MAX = 1 (effectively VERP) and any kind of batching (SMTP > 1) is roughly 50%. To get VERP or customized footers or customized anything, you double your network bandwidth. There is very little advantage to setting SMTP_MAX > 5, UNLESS your subscriber base is heavily stratified onto very few sites. If you have really large groups of subscribers on AOL or Hotmail, it can help cut network bandwidth, but at best, it seems to be about a 10% improvement. If you plot the numbers I did on a curve, you can see just how little advantage you get by increasing the number. You get almost all of the advantage by going to 2, and the line past 5 is very flat.... Interesting -- I honestly didn't expect to see THIS big a difference -- I was expecting more like 25-30% increase in bandwidth for a VERP-type delivery. My thoughts on what this means to future directions: Customized messages (VERPing, or encoded unsub URLs, or all of that...) should definitely be an option in Mailman 2.1. I would set Mailman's 2.1 default to have this turned ON, giving us the customized unsub links and etc, but to document this for users so they know to turn it off on slow networks. If users turn it off, I recommend that SMTP_MAX be set by default to 5, and that we document that it makes little sense to change it unless a site is horribly network limited, because even setting to the max only gains them another 10% (and if they're THAT network limited, they're seriously asking for trouble anyway), and only if their subscriber base fits a profile that lends itself to the compression. Setting it large also leaves them open to spamblocking by systems that don't necessarily follow the standards or act right, too. We should ALSO note here that some MTAs (postfix, for instance) might override SMTP_MAX anyway -- you could set it to 100, but postfix might be configured smaller, so they have to be aware of those potential interactions. you then get into the issues of tuning all this, with few delivery threads with lots of addresses vs many threads in parallel.. and all that fun -- I guess I'm trying to say that you can't tune mailman in isolation from the MTA (and down that road lies a huge rathole of attempting to document this stuff...) But from these numbers, any 2.0.x version of mailman should set SMTP_MAX to between 2 and 5, unless they're horribly network limited. it makes no sense to be larger than 5, and it makes no sense to be 1 unless you've done some kind of VERPing patch. for 2.1, we want to implement these customizations and default them on, but with a 50% network hit, we definitely want to make it clear what's going on and make it possible for them to turn it off and return to a generic URL and non-customized e-mail. Barry's mileage may vary on his preferences for default, of course, and it's his show. I think the advantages of the customized URL/email capability is a huge one and most sites will benefit from it -- but the network hit might kill some sites, so we have to give them an easy ability to turn the feature off. What do y'all think? I've included mailman-developers on this reply, since while this started on mm-users, it really ought to be discussed on the developers list... -- Chuq Von Rospach, Internet Gnome <http://www.chuqui.com> [<chuqui(a)plaidworks.com> = <me(a)chuqui.com> = <chuq(a)apple.com>] Yes, yes, I've finally finished my home page. Lucky you. Yes, I am an agent of Satan, but my duties are largely ceremonial.

3 4

Re: [Mailman-Developers] Braindump on new archiver
by J C Lawrence 16 Jun '01

16 Jun '01

On Sat, 16 Jun 2001 13:34:02 -0400 (EDT) Dale Newfield <Dale(a)Newfield.org> wrote: > On Sat, 16 Jun 2001, Thomas Wouters wrote: >> Maildir works by making every message a file on its own. A >> mailbox is a directory with three subdirectories, 'cur', 'new' >> and 'tmp'. Messages in 'new' are unread, messages in 'cur' are >> read, and messages in 'tmp' are in transit. > What about file system limitations to the maximum number of files > in a directory? Typically this is an inode limit for the whole filesystem, not a directory size issue. Directory size is a performance issue that different filesystems handle variously well (eg Ext2FS badly, ReiserFS fairly well). -- J C Lawrence claw(a)kanga.nu ---------(*) http://www.kanga.nu/~claw/ The pressure to survive and rhetoric may make strange bedfellows

2 1

Braindump on new archiver
by Thomas Wouters 16 Jun '01

16 Jun '01

I've been thinking about re-doing the whole archiver thing in Mailman for a while now, and I need to start writing these things down before I forget about them. What follows is basically a braindump, for your amusement and free to comment on. First off, I have a couple of handicaps. Some are severe. The first one is that I have very little time, which is one of the reasons I'm writing this braindump: I don't have time to implement it right now :P The other is that, for a living, I build and maintain large to very large, high-performance internet servers (mostly Apache webservers with bells 'n whistles.) This means I am usually too performance oriented. And lastly: I like the 'Maildir' mailbox format very much. This is a handicap because, unfortunately, not all mail programs support it. 'Pine', specifically, is boycotting the Maildir format (don't ask me why, I gave up on them.) The good thing about the Maildir format is that it's very NFS secure. All of our customers (except a handful who refuse to upgrade from 'elm' to 'mutt') have a maildir mailbox, and we have 6 SMTP delivery boxes, 5 POP retrieval boxes, one IMAP retrieval box (used only for our webmail server, which is why it's only one right now) and several shell servers that have all kinds of UNIX mail clients running. The mailspool itself is a dedicated NFS server (a NetApp Filer.) We have a patched version of Pine that works with maildir whether it likes it or not :) Maildir works by making every message a file on its own. A mailbox is a directory with three subdirectories, 'cur', 'new' and 'tmp'. Messages in 'new' are unread, messages in 'cur' are read, and messages in 'tmp' are in transit. Message-metadata and state is maintained using the name of the message-file itself. Delivery is done by creating a new file in mailbox/tmp/, writing all data to it, and when it's done, moving it to 'new'. Writing data to a file is not trustworthy, over NFS, but renaming and moving files are. What I'm thinking for a Mailman archiver (or just any email-webarchiver, in fact) is something like the following: - Mail delivery can happen on any host. The Mailman process, after doing its normal pre-delivery stuff, 'delivers' the message by dropping it in a Maildir folder (rather than appending it to an 'mbox' mailbox) _and_ cgi-escaping/header-cleaning the message and dropping it in a monthly/weekly/daily archive directory. The names are generated from uniq data, like the time of day, the machine name and the process-id, possibly hashed for obscurity. No locking is necessary. The URL to that _specific_ message is known as soon as Mailman calculates that name, and so it can be inserted into the message that will be sent out to the list. (A common feature request :) - Non-text attachements could be saved as separate files, and replaced by URLs to them in the archives. This should be possible to do both based on type and on size, of course. - The web archives are indexed once every [X] minutes, by looking at all the new messages and processing them. Processing them consists of creating indexes based on threads, subjects, author, etc, exactly like now. State data could be stored in a .db (berkeley db, marshall, pickle, whatever) and need not even be protected by a lock, if done properly: When started, the indexing process should compare the mtime of the .db file with the current time, as well as the last added new message. If the .db file is too new, it should not run. If the .db file is old, and the newest message is newer, it should touch the .db file (leaving it exactly as it is, just updating mtime) and start working, keeping in mind to touch the file every now and then. The new .db file should be written to a temporary name, and rename()'d when it's done. - The individual archive message URL's would actually call a CGI script that looks up state data from the .db file, grab the cgi-escaped file, prepend and append standard header and footer containing 'next in thread', 'next in subject' and the like generated from the .db file, and feed that to the browser. If state data is absent, but the message itself is there, teh standard header and footer are added with a note saying that the message is too new to have links to other messages -- but at least the message will be visible. Mail delivery and the Web-interface can be run on multitudes of servers, all sharing data over NFS, and still be able to communicate in a sane way, without waiting for eachother's locks forever. The only thing that can go wrong is for a message to be archived and sent out, and then read and looked up in the archives before the actual NFS data (the content of the message) is visible on the web-interface machine(s). Given the average speed of SMTP, I do not consider this a serious problem :) One thing should run on a single server, however, and that's the archive-indexing process. Given that news-feeding and mailing out passwords should also be done on a single server, I don't see that as a problem either. And all of the above should work just as fine on a single machine doing everything on a local disk. While under the 'the filesystem as the database' spell (hell, I'm still under that one) I also considered using the filesystem to index messages based on thread, subject, author, etc: a directory for each category, in which symlinks point to messages. To go from message 'X' to the next message in the thread, check where the symlink 'next-threads/X' points. The same for previous-thread, next-author, etc. I'm not sure if it will managable without adding a lockfile, nor how efficient it would be to generate overviews from such data. Another thing for consideration is the 'downloadable mailbox' link. Should it download the maildir mailbox, or generate an mbox one on demand ? The maildir mailbox is *very* useful for incremental updates, since all that takes is checking which files you've already read. But, like I said, it's not as widely supported as the crappy ol' mbox format. End-of-rant--we-will-now-return-to-your-regular-scheduled-static-ly y'rs, ;) -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

2 1

Delayed Mail (still being retried) (fwd)
by Carson Gaspar 16 Jun '01

16 Jun '01

Attached is one of the postfix messages that the bounce handler misses. -- Carson Gaspar - carson(a)gnac.com Security Architect - Global Networking and Computing, Inc. This is the Postfix program at host mail.knoware.nl. #################################################################### # THIS IS A WARNING ONLY. YOU DO NOT NEED TO RESEND YOUR MESSAGE. # #################################################################### Your message could not be delivered for 4 hours. It will be retried until it is 5 days old. For further assistance, please contact <postmaster(a)knoware.nl> The Postfix program --- Delivery error report follows --- <bashakst(a)knoware.nl>: temporary failure. Command output: bashakst: Over quota _ --- Undelivered message follows --- Received: from imail.ision.nl (imail.ision.nl [195.64.48.18]) by mail.knoware.nl (Postfix) with ESMTP id 8B71E39740 for <bashakst(a)knoware.nl>; Fri, 15 Jun 2001 20:44:19 +0200 (CEST) Received: from lists.gnac.net (pluto.gnac.com [209.182.195.144]) by imail.ision.nl (Postfix) with ESMTP id 4913626632 for <bashakst(a)knoware.nl>; Fri, 15 Jun 2001 20:44:19 +0200 (CEST) Received: from lists.gnac.net (localhost [127.0.0.1]) by lists.gnac.net (Postfix) with ESMTP id 8F270106E4; Fri, 15 Jun 2001 11:40:00 -0700 (PDT) Delivered-To: firewalls(a)lists.gnac.net Received: from femail3.rdc1.on.home.com (femail3.rdc1.on.home.com [24.2.9.90]) by lists.gnac.net (Postfix) with ESMTP id 11BDD106AD for <firewalls(a)lists.gnac.net>; Fri, 15 Jun 2001 11:30:28 -0700 (PDT) Received: from Jason ([24.42.170.206]) by femail3.rdc1.on.home.com (InterMail vM.4.01.03.20 201-229-121-120-20010223) with SMTP id <20010615183025.FVHI15293.femail3.rdc1.on.home.com@Jason> for <firewalls(a)lists.gnac.net>; Fri, 15 Jun 2001 11:30:25 -0700 From: "Jason Brown" <jason(a)onsidekick.com> To: <firewalls(a)lists.gnac.net> Subject: Network Sniffers Message-ID: <000f01c0f5c9$ab247a20$6500a8c0(a)onside.tor> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: firewalls-admin(a)lists.gnac.net Errors-To: firewalls-admin(a)lists.gnac.net X-BeenThere: firewalls(a)lists.gnac.net X-Mailman-Version: 2.0.5 Precedence: bulk List-Help: <mailto:firewalls-request@lists.gnac.net?subject=help> List-Post: <mailto:firewalls@lists.gnac.net> List-Subscribe: <http://lists.gnac.net/mailman/listinfo/firewalls>, <mailto:firewalls-request@lists.gnac.net?subject=subscribe> List-Id: Firewalls <firewalls.lists.gnac.net> List-Unsubscribe: <http://lists.gnac.net/mailman/listinfo/firewalls>, <mailto:firewalls-request@lists.gnac.net?subject=unsubscribe> List-Archive: <http://lists.gnac.net/pipermail/firewalls/> Date: Fri, 15 Jun 2001 14:33:26 -0400

1 0

Postfix DSN handling
by Carson Gaspar 16 Jun '01

16 Jun '01

mailman 2.0.5 does not handle Postfix DSN messages for delayed mail. Has this been fixed in the development tree, or should I attempt to patch the code myself? -- Carson Gaspar - carson(a)taltos.org Queen trapped in a butch body

2 1

Configuring mailman with the scripts?
by Danny Graham 15 Jun '01

15 Jun '01

I am unable to find the config files on the server that is hosting me. They would probably not appreciate it if I changed them anyways :) Is there anyway for me to customize and change the welcome message to subscribers? Basically I just want to give them a URL to unsubscribe at and remove the info about how to post to the list... Help would be TOTALLY appreciated. -- Cheers - Danny Graham ---------------------------------------------------------------------------- ---- //liberate possibility::. --MIND SQUEEZE INTERACTIVE //:: ---------------------------------------------------------------------------- ---- //liberate possibility::. --MIND SQUEEZE INTERACTIVE //::

1 0

Configuring mailman without the scripts?
by Danny Graham 15 Jun '01

15 Jun '01

I am unable to find the config files on the server that is hosting me. They would probably not appreciate it if I changed them anyways :) Is there anyway for me to customize and change the welcome message to subscribers? Basically I just want to give them a URL to unsubscribe at and remove the info about how to post to the list... Help would be TOTALLY appreciated. -- Cheers - Danny Graham -------------------------------------------------------------------------------- //liberate possibility::. --MIND SQUEEZE INTERACTIVE //::

1 0

anonymity features
by Yanuar Nugroho 15 Jun '01

15 Jun '01

Dear folks, I'm doing a research on "how anonymity in CMC influences communication behaviour among participants/communicators". This will be done in asynchronous mode, and I have chosen to use GNU Mailman rather than any other systems. This is the list of the "anonymity feature" requirements that should be ideally able to include. 1. Removing the sender identities (it has been solved, i think. in "privacy" setting and cleanse.py, right?) 2. Archiving sent message (how is actually a message archived in Mailman?) 3. Adding additional text from a database into a message before being distributed (is it possible to create text-based database and include a particular data to the message? how can Mailman query and retrieve a data to be included -- how can Mailman "scan" a message and find a "keyword" and use the keyword to retrieve relevant data? 4. Other idea for the no.3 above, using "phantom messages" as the database. The problem is how can Mailman automatically add a phantom message to a message? 5. How to customise the text-wrapping (text area in the email) in Mailman? How to set, for example, the LEFT Margin to be 5 characters? I think I have to stop. It has been too much. Sorry-sorry-sorry very much if this realy bothers you. I am desperate in doing this research ... :) I don't know how to do this. Not even a clue. I'm totally a new comer in Mailman, Python and this GNU environment, but I want to learn. Anyone can help me? (thanks to Barry Warsaw who encouraged me to join this milist. also for Yaron Zarfati, Detlef Neubauer & JC Lawrence who tried to solve my previous problems in handling the header. thanks!:)). Cheers /Yanuar. PS. If any of you need further info, I can send my preliminary works/report via your individual emails. This probably helps you to know the context of my research. Yanuar Nugroho http://yanuar-nugroho.mainpage.net http://www.yanuar-n.net ---------------------------------------- PG. Stud. MSc. Information Systems Engineering, Dept. of Computation, UMIST, Manchester - UK

1 0

Re: auto aprroval
by Mark Tearle 15 Jun '01

15 Jun '01

On Tue, 12 Jun 2001, Marcin Bujak wrote: > hi > i'm having a problem with your patch "Auto Approval of > subscriptions for certain domains" > > cd mailman-2.0.5/Mailman > patch < None.txt > cd .. > make clean > make install > > if i choose a privace options from privacy options section > mailman send a bug in .... > > can you help me ? > Hiya The patch is buggy and I should really get around to updating the Sourceforge web page about it. It works OK if you create a new list but doesn't detect if the field is missing in an existing list and add it unfortunately. I'm CC'ing mailman-developers to ask for advice, I know it goes somewhere in the versioning code, but where? And how do I do it such that future upgrades etc don't break? Yours Mark -- Mark Tearle - mark(a)tearle.com "The only good is knowledge and the only evil is ignorance." - Socrates

2 1