Mailman 3 January 2003 - Mailman-Developers

Bounce removal parameters default values
by Greg Stark 23 Oct '03

23 Oct '03

I find I am being removed from mailman mailing lists left and right. I believe the default values for the bounce removal should be reconsidered. It's possible that you haven't had many users in my situation and so haven't really had a chance to tune these parameters on the low end yet. But they clearly aren't working for me at a few sites. My particular situation is that my site has seen fit to filter viruses by refusing delivery. This causes a bounce from the remote MTA every time someone sends me an Outlook virus. Why my site administrators felt this was necessary is a question for another day, it's not like I use Outlook or like my spam filters wouldn't have thrown these messages away anyways, but whatever. The net result is that some small fraction of messages to me bounce and list management software notices this. The only reason I became aware of the problem was because ezmlm also does this type of processing but it sends a warning message before removing users. It only removes you if the warning message itself bounces. In fact it sends two such warning messages and only removes the user if *both* bounce. This provides the user with a chance to react to the first message and fix the problem -- if they ever see the message. I could beg for a similar feature in mailman, but I'm not sure it's necessary. But I am sure it's necessary to tune the bounce processing parameters. The relatively few bounces I'm generating shouldn't be causing me to get removed when all the real messages are being delivered fine. It seems the legitimate messages that are correctly delivered should reset the count of bounces to 0. Reading the source it seems it has to see DEFAULT_MAX_POSTS_BETWEEN_BOUNCES such legitimate posts between messages. I'm fairly convinced this parameter should always be 0. If any successful delivery occurs the user should never be removed due to bounces. What I don't understand is how DEFAULT_MAX_POSTS_BETWEEN_BOUNCES relates to the parameters I see in the admin. None of the parameters in the admin corresponds to this. How is it calculated? -- greg

6 15

Re: [Mailman-Developers] Mailman password and administrative reminders
by Dan Mick 14 Aug '03

14 Aug '03

? This very message came to me with the following header: Errors-To: mailman-developers-admin(a)python.org All my bounces come to the list admin address, set in the admin webpage, in the second field of General Options. Do you have that set to something, and bounces still come to root? > 2. Bounces are sent to the poor postmaster instead of a -admin address. > I'm not entirely certain, but I think an Errors-To: header or something > like that in all Mailman messages might allow one to distribute that load > somewhat.

8 11

english-only.patch
by Matthew Davis 31 Jan '03

31 Jan '03

http://sourceforge.net/tracker/index.php?func=detail&aid=677940&group_id=10… Thats the first attempt to make an english-only version of mailman (no offense to the multi-lingual users). But it takes up a lot of space. Any suggestions/comments/help/pointers would be appreciated. With all the languages. -rw-r--r-- 1 root root 5466939 Jan 28 21:49 mailman-2.1.tgz After patch and taking out the languages. -rw-rw-r-- 1 mailman3 mailman3 2357419 Jan 30 22:50 mailman-2.1-en.tgz As you can see its less than half the size. Now my question is how would be a good way to put the languages back in? Package them in a tarball with a patch? And make a tarball for each language? -- Matthew Davis http://dogpound.vnet.net/ All I need to know I learned from my cat.

2 1

[Mailman-Developers] My Scroll To CMM Needs Help To Work With All Mailman Digests
by Bernie Zenis 31 Jan '03

31 Jan '03

Hi, I'm looking for what is constant in all Mailman digests. For example, - Do all digests start with "Send <list name> mailing list submissions to"? - Do all digests have a "Today's Topics" section? - Do all messages start with "Message: <message number>"? - Do all digests end with "End of <list name> Digest"? I'm working on a Contextual Menu Module/Item/Plugin/whatever-Apple-calls-it-today for Mac OS X. It allows the user to Control-Click in a mailman digest and get a "Scroll To" submenu which allows easy scrolling directly to any message. I originally wrote the code to work with digests of Apple's mailing lists (http://www.lists.apple.com/). Now I see that Apple and many others use Mailman. So, I decided to try to generalize my code to work with all Mailman digests. I don't know Mailman or anything about it. I need some help so that my code can robustly scan Mailman digests. I realize that I will probably have to work with text in non-English languages if I want my CMM to work with non-English digests. Thanks, Bernie

1 0

E-mail validation code
by Charlie Clark 31 Jan '03

31 Jan '03

Dear list, I've recently had a whole load of e-mail addresses to check and have been looking for a way to help this. I was pointed in the direction of Mailman/Utils.py:ValidateEmail by Danny Yoo on the Python tutor list. It looks like this: ### _badchars = re.compile('[][()<>|;^,]') def ValidateEmail(str): """Verify that the an email address isn't grossly invalid.""" # Pretty minimal, cheesy check. We could do better... if not str: raise Errors.MMBadEmailError if _badchars.search(str) or str[0] == '-': raise Errors.MMHostileAddress if string.find(str, '/') <> -1 and \ os.path.isdir(os.path.split(str)[0]): # then raise Errors.MMHostileAddress user, domain_parts = ParseEmail(str) # this means local, unqualified addresses, are no allowed if not domain_parts: raise Errors.MMBadEmailError if len(domain_parts) < 2: raise Errors.MMBadEmailError ### My unskilled eye agrees with the comment. Having trawled the web alternatives I've come across the following approaches: The Python stuff is at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66439/index_txt It is a module with a single class which comes with some built in patterns and the ability to make custom patterns. The built in e-mail test doesn't catch things like spaces or @@ and other things. The perl stuff is at http://aspn.activestate.com/ASPN/Cookbook/Rx/Recipe/68432/index_txt sub ValidEmailAddr { #check if e-mail address format is valid my $mail = shift; #in form name@host return 0 if ( $mail !~ /^[0-9a-zA-Z\.\-\_]+\(a)[0-9a-zA-Z\.\-]+$/ ); #characters allowed on name: 0-9a-Z-._ on host: 0-9a-Z-. on between: @ return 0 if ( $mail =~ /^[^0-9a-zA-Z]|[^0-9a-zA-Z]$/); #must start or end with alpha or num return 0 if ( $mail !~ /([0-9a-zA-Z]{1})\@./ ); #name must end with alpha or num return 0 if ( $mail !~ /.\@([0-9a-zA-Z]{1})/ ); #host must start with alpha or num return 0 if ( $mail =~ /.\.\-.|.\-\..|.\.\..|.\-\-./g ); #pair .- or -. or -- or .. not allowed return 0 if ( $mail =~ /.\.\_.|.\-\_.|.\_\..|.\_\-.|.\_\_./g ); #pair ._ or -_ or _. or _- or __ not allowed return 0 if ( $mail !~ /\.([a-zA-Z]{2,3})$/ ); #host must end with '.' plus 2 or 3 alpha for TopLevelDomain (MUST be modified in future!) return 1; } This seems to catch pretty much everything but it's perl and I'm not sure what !~ and =~ do I've started work on making custom definitions based on the perl source like this sv1 = StringValidator("joe(a)testmail.com") sv1.definePattern("test1", "^[0-9a-zA-Z\.\-\_]+\(a)[0-9a-zA-Z\.\-]+$") sv1.definePattern("test2", "^[^0-9a-zA-Z]|[^0-9a-zA-Z]$") if not sv1.isValidForPattern("test1"): print sv1.validateString, " has invalid characters in the name" elif not sv1.isValidForPattern("test1"): print sv1.validateString, " doesn't start or end with alpha or num" else: print sv1.validateString, "is valid" These tests work pretty well but I'm having trouble turning the perl lines with =~ into usable Python code: they tend to invalidate real addresses. And I've been given http://www.interclasse.com/scripts/EMailValidatorCLS.php I'll confess to not being much of a programmer but I'm sure I can come up with an improvement on the current function with a little help. Thank you. Charlie -- Charlie Clark Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-938-5360

1 0

[Mailman-Developers] BounceRunner optimization and problems with VERY LARGE lists
by John 31 Jan '03

31 Jan '03

Hi, I'm new to the list and have been using Mailman 2.1 for about month on a single list of 138,000 subscribers (It's a legit opt-in announce only list for my wife's website that's taken 3 years to grow to this size). A typical mailing of this list generates over 5,000 bounces, giving BouncerRunner A LOT of work. At best, BounceRunner can process these at 20/minute on my 2GHZ P4 Redhat mail server, taking over 4 hours best case to finish (running at CPU 90% utilization). However, when it is doing all this work I have the following problems: (1) The Web gui Membership management page times out and is unusable (2) If CommandRunner starts processing commands I get "couldn't get list lock" errors in BounceRunner's log (Of course losing the Bounce updates to the list) After studying the code in the Runner class (and BounceRunner in particular) I believe I have a solution to these problems but I wanted to get a sanity check from everyone on the list BEFORE beginning rewriting the code. Here is a greatly simplified overview of how I understand BounceRunner currently processes bounces Mailman V 2.1 code: I've highlighted the troublespots in CAPS While Forever (Process all the emails we find in the bounce queue) For Every email in queue REREAD list from disk Dequeue the message Extract addresses to bounce LOCK the LIST For Every address in message Register Bounce SAVE the list to disk UNLOCK the list If we didn't PROCESS ANY EMAILS on last pass Then SLEEP for SLEEPTIME CLEANUP ON EXIT I believe that these are the troublespots that have been causing me problems: (1) The list SAVE is executed once for every bounced email. For my big list, that's 13 Megabytes of data written and read back from the disk for EVERY bounce email. Which is why it takes 2-3 seconds to process an email. (2) BounceRunner is VERY greedy about the list lock. The time "window" for other processes to acquire a list lock is VERY short when the bounce queue is filling or full. In this case, the lock is only open for the time it takes to extract the addresses from the next email! In additon, because we ONLY sleep when the QUEUE is empty this behavior can exist for HOURS on a large list. Here's my version of the new improved BounceRunner intialize x to number of bounces to process on each pass While Forever Initialize Python list structure to hold bounces (Process x emails in the bounce queue) For x emails in queue Dequeue the message Extract addresses to bounce SAVE address and Listname in Python list structure If Python List structure contains emails For all mailing lists in Python structure REREAD list from disk LOCK the LIST For all addresses that bounced for this list Register Bounce SAVE the list to disk UNLOCK the list SLEEP for SLEEPTIME CLEANUP on exit Advantages to this method: (1) We process a number of bounces before writing out the list reducing I/O (the real bootleneck) by factor x. When x is one the algorithm almost degenerates to the current method (2) Since we always sleep on each pass it gives other processes (like the Web gui) a chance to read the list. (3) By increasing x we control the number of bounces that get processed on each pass. The time it takes to extract the addresses gives other processes time to acquire the list lock and avoid "lockout" (4) Since "in memory" bounce registration is very fast we can do a lot of them while the list is locked without adding significantly to the already long lock time on a big list (I believe the I/O is the limiting factor) Disadvantages: (1) A larger number of bounces could be lost if we can't acquire the list lock to update the list. If desired, we could write the extracted addresses to a file to allow easier recovery in this situation. However, since they are just bounces it's not a huge loss anyway. (2) The processing time for the larger number of bounces WILL be greater than the single bounce processed now. How much more I don't know. This will mean that the list will be locked for a longer period on each pass. However, it will be locked LESS frequently since the bounces can be cleared from the queue faster. I'm thinking of a similar strategy for CommandRunner, since that is my other resource hog, taking 2-5 seconds per subscribe or unsubscribe. Thoughts? Comments? Suggestions? I'm interested in any and all responses.

1 0

Notes on MM on OS X 10.1 vs 10.2
by Stonewall Ballard 29 Jan '03

29 Jan '03

I finally updated my server to OS X Jaguar Server, which fixed both problems I had with Mailman. For the record, both the problem with the missing "strptime" and the configure problem with the mailman user being untestable are gone on OS X 10.2. - Stoney -- Stonewall Ballard stoney(a)sb.org http://stoney.sb.org/

1 0

Quick introduction and a few questions
by Charlie Clark 29 Jan '03

29 Jan '03

Hi, I'm a long time user and champion of mailman and have joined this list as it looks like my next project will require mailman with changes. I am working on a newsletter system (publish only) which is multi-channel: web, e-mail, SMS, MMS, fax, etc. There are currently over 10,000 users. I am hoping to use mailman for the e-mail part but hopefully obvious reasons would like to keep user management outside of mailman. I imagine this is a common need. I have searched the site checked Google but there doesn't seem to be much about apart from a proposal for an InterfaceTemplate on Barry's page at Zope.org. Questions: is this already possible? and if so where do I get more info? if this is not already possible who else has got a similar need? I'm also looking at QoS for e-mail delivery, ie. make sure all e-mails leave their MTAs within 15 of posting. From what I've read on the FAQs mailman seems to be predestined to implement this provide there sufficient MTAs which are correctly configured. What suggestions for metrics in this regard do people have. Thanx

1 0

poor choice of words on list overview page
by Martin Pool 29 Jan '03

29 Jan '03

In Mailman 2.1, the cgi which gives the list of available mailing lists says something like this: > Below is a listing of all the public mailing lists on > lists.samba.org. Click on a list name to get more information about > the list, or to subscribe, unsubscribe, and change the preferences on > your subscription. To visit the general information page for an > unadvertised list, open a URL similar to this one, but with a '/' and > the list name appended. > List administrators, you can visit the list admin overview page to > find the management interface for your list. > Send questions or comments to mailman(a)lists.samba.org A few people persistently interpret the "send questions or comments" as meaning "send samba user questions" rather than "send questions or comments about the mailing lists". This seems to happen a couple of times a week. (Fairly reliably the people who make this mistake also send mail that is lacking in any useful detail and is therefore pretty pointless anyhow, but nevertheless they have to be forwarded or bounced.) I think a better phrasing would be something like "Please send problem reports about the mailing list to %s" Ideally this text might be a per-site customization, rather than hardcoded in Mailman/Cgi/listinfo.py. I have fixed it there for the time being. -- Martin

1 0

[Mailman-Developers] bug? "already subscribed" message for disabled accounts
by Martin Pool 29 Jan '03

29 Jan '03

Mailman 2.1 (which rocks in most respects) gives a message like this: > An attempt was made to subscribe your address to the mailing list > foo(a)bar.org. You are already subscribed to this mailing list. I agree with this in general, but it seems that it would be better to detect the case where the user has deliveries disabled. I'm sure some people will find this puzzling: they think they're unsubscribed, but Mailman says they are. Perhaps the message should conditionally tell them they have delivery disabled, and that it can be reenabled through the web interface. Cheers -- Martin

1 0