Mailman-Developers September 2001

mailman-developers@python.org

35 participants
58 discussions

Re: [Mailman-Developers] Mailman password and administrative reminders

by Dan Mick

? This very message came to me with the following header: Errors-To: mailman-developers-admin(a)python.org All my bounces come to the list admin address, set in the admin webpage, in the second field of General Options. Do you have that set to something, and bounces still come to root? > 2. Bounces are sent to the poor postmaster instead of a -admin address. > I'm not entirely certain, but I think an Errors-To: header or something > like that in all Mailman messages might allow one to distribute that load > somewhat.

18 years, 2 months

[ANNOUNCE] Mailman 2.1 alpha 2

by barry＠digicool.com

This the official announcement for Mailman 2.1 alpha 2. Because it's an alpha, this announcement is only going out to the mailman-* mailing lists. I'll make two warnings: you probably should still not use this version for production systems (but TIA for any and all testing you do with it!), and I've already had a couple of bug fixes from early adopters. 2.1a2 should still be useful, but you might want to keep an eye on cvs and the mailman-checkins list for updates. I am only making the tarball available on SourceForge, so you'll need to go to http://sf.net/projects/mailman to grab it. You'll also need to upgrade to mimelib-0.4, so be sure to go to http://sf.net/projects/mimelib to grab and install that tarball first. To view the on-line documentation, see http://www.list.org/MM21/index.html or http://mailman.sf.net/MM21/index.html Below is an excerpt from the NEWS file for all the changes since 2.1alpha1. There are a bunch of new features coming down the pike, and I hope to have an alpha3 out soon. I'm also planning on doing much more stress testing of this version with real list traffic, and I'm hoping we'll start to get more languages integrated into cvs. Enjoy, -Barry -------------------- snip snip -------------------- 2.1 alpha 2 (11-Jul-2001) - Building o mimelib 0.4 is now required. Get it from http://mimelib.sf.net. If you've installed an earlier version of mimelib, you must upgrade. o /usr/local/mailman is now the default installation directory. Use configure's --prefix switch to change it back to the default (/home/mailman) or any other installation directory of your choice. - Security o Better definition of authentication domains. The following roles have been defined: user, list-admin, list-moderator, creator, site-admin. o There is now a separate role of "list moderator", which has access to the pending requests (admindb) page, but not the list configuration pages. o Subscription confirmations can now be performed via email or via URL. When a subscription is received, a unique (sha) confirm URL is generated in the confirmation message. Simply visiting this URL completes the subscription process. o In a similar manner, removal requests (via web or email command) no longer require the password. If the correct password is given, the removal is performed immediately. If no password is given, then a confirmation message is generated. - Internationalization o More I18N patches. The basic infrastructure should now be working correctly. Spanish templates and catalogs are included, and English, French, Hungarian, and Big5 templates are included. o Cascading specializations and internationalization of templates. Templates are now search for in the following order: list-specific location, domain-specific location, site-wide location, global defaults. Each search location is further qualified by the language being displayed. This means that you only need to change the templates that are different from the global defaults. Templates renamed: admlogin.txt => admlogin.html Templates added: private.html - Web UI o Redesigned the user options page. It now sits behind an authentication so user options cannot be viewed without the proper password. The other advantage is that the user's password need not be entered on the options page to unsubscribe or change option values. The login screen also provides for password mail-back, and unsubscription w/ confirmation. Other new features accessible from the user options page include: ability to change email address (with confirmation) both per-list and globally for all list on virtual domain; global membership password changing; global mail delivery disable/enable; ability to suppress password reminders both per-list and globally; logout button. [Note: the handle_opts cgi has gone away] o Color schemes for non-template based web pages can be defined via mm_cfg. o Redesign of the membership management page. The page is now split into three subcategories (Membership List, Mass Subscription, and Mass Removal). The Membership List subcategory now supports searching for member addresses by regular expression, and if necessary, it groups member addresses first alphabetically, and then by chunks. Mass Subscription and Mass Removal now support file upload, with one address per line. o Hyperlinks from the logos in the footers have been removed. The sponsors got too much "unsubscribe me!" spam from desperate user of Mailman at other sites. o New buttons on the digest admin page to send a digest immediately (if it's non-empty), to start a new digest volume with the next digest, and to select the interval with which to automatically start a new digest volume (yearly, monthly, quarterly, weekly, daily). DEFAULT_DIGEST_VOLUME_FREQUENCY is a new configuration variable, initially set to give a new digest volume monthly. o Through-the-web list creation and removal, using a separate site-wide authentication role called the "list creator and destroyer" or simply "list creator". If the configuration variable OWNERS_CAN_DELETE_THEIR_OWN_LISTS is set to 1 (by default, it's 0), then list admins can delete their own lists. This feature requires an adaptor for the particular MTA you're using. An adaptor for Postfix is included, as is a dumb adaptor that just emails mailman@yoursite with the necessary Sendmail style /etc/alias file changes. Some MTAs like Exim can be configured to automatically recognize new lists. The adaptor is selected via the MTA option in mm_cfg.py - Email UI o In email commands, "join" is a synonym for "subscribe". "remove" and "leave" are synonyms for "unsubscribe". New robot addresses are support to make subscribing and unsubscribing much easier: mylist-join@mysite mylist-leave@mysite o Confirmation messages have a shortened Subject: header, containing just the word "confirm" and the confirmation cookie. This should help for MUAs that like to wrap long Subject: lines, messing up confirmation. o Mailman now recognizes an Urgent: header, which, if it contains the list moderator or list administrator password, forces the message to be delivered immediately to all members (i.e. both regular and digest members). The message is also placed in the digest. If the password is incorrect, the message will be bounced back to the sender. - Performance o Refinements to the new qrunner subsystem which preserves FIFO order of messages. o The qrunner is no longer started from cron. It is started by a Un*x init-style script called bin/mailmanctl (see below). cron/qrunner has been removed. - Command line scripts o bin/mailmanctl script added, which is used to start, stop, and restart the qrunner daemon. o bin/qrunner script added which allows a single sub-qrunner to run once through its processing loop. o bin/change_pw script added (eases mass changing of list passwords). o bin/update grows a -f switch to force an update. o bin/newlang renamed to bin/addlang; bin/rmlang removed. o bin/mmsitepass has grown a -c option to set the list creator's password. The site-wide `create' web page is linked to from the admin overview page. o bin/newlist's -o option is removed. This script also grows a way of spelling the creation of a list in a specific virtual domain. o The `auto' script has been removed. o bin/dumpdb has grown -m/--marshal and -p/--pickle options. o bin/list_admins can be used to print the owners of a mailing list. o bin/genaliases regenerates from scratch the aliases and aliases.db file for the Postfix MTA. - Archiver o New archiver date clobbering option, which allows dates to only be clobber if they are outrageously out-of-date (default setting is 15 days on either side of received timestamp). New configuration variables: ARCHIVER_CLOBBER_DATE_POLICY ARCHIVER_ALLOWABLE_SANE_DATE_SKEW The archived copy of messages grows an X-List-Received-Date: header indicating the time the message was received by Mailman. o PRIVATE_ARCHIVE_URL configuration variable is removed (this can be calculated on the fly, and removing it actually makes site configuration easier). - Miscellaneous o Several new README's have been added. o Most syslog entries for the qrunner have been redirected to logs/error. o On SIGHUP, qrunner will re-open all its log files and restart all child processes. See "bin/mailmanctl restart". - Patches and bug fixes o SF patches and bug fixes applied: 420396, 424389, 227694, 426002, 401372 (partial), 401452. o Fixes in 2.0.5 ported forward: Fix a lock stagnation problem that can result when the user hits the `stop' button on their browser during a write operation that can take a long time (e.g. hitting the membership management admin page). o Fixes in 2.0.4 ported forward: Python 2.1 compatibility release. There were a few questionable constructs and uses of deprecated modules that caused annoying warnings when used with Python 2.1. This release quiets those warnings. o Fixes in 2.0.3 ported forward: Bug fix release. There was a small typo in 2.0.2 in ListAdmin.py for approving an already subscribed member (thanks Thomas!). Also, an update to the OpenWall security workaround (contrib/securelinux_fix.py) was included. Thanks to Marc Merlin.

19 years, 7 months

Internal server error when accessing admin pages

by Marc MERLIN

I've gotten several bug reports on SF for people who can't get to the admin pages The only error apache gives me is: [Thu Jul 12 15:47:17 2001] [error] [client 171.71.41.31] Premature end of script headers: /var/local/mailman/cgi-bin/admin I never had the problem myself and it seems to come and go Is there any debugging I can in mailman to at least give me some error in my apache error log? (or do you think it's an apache problem?) Attached is one of the tickets I've gotten Thanks, Marc ----- Forwarded message from noreply(a)sourceforge.net ----- To: noreply(a)sourceforge.net From: noreply(a)sourceforge.net Subject: [ alexandria-Support Requests-438992 ] Administration interface to mail lists Date: Thu, 26 Jul 2001 02:43:44 -0700 Support Requests item #438992, was opened at 2001-07-06 00:49 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=200001&aid=438992&group_id… Category: Mailing Lists Group: Second Level Support >Status: Open Priority: 9 Submitted By: Richard Leyton (rleyton) Assigned to: Marc Merlin (marcmerlin) Summary: Administration interface to mail lists Initial Comment: A post to a mailing list needs admin attention (it's a moderated list). The URL: http://lists.sourceforge.net/lists/admindb/leap-announce results in: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, staff(a)sourceforge.net and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Could you look into it. ---------------------------------------------------------------------- >Comment By: Richard Leyton (rleyton) Date: 2001-07-26 02:43 Message: Logged In: YES user_id=7704 In a nutshell, this ceased to be a problem the next day, i'd forgotten about this ticket until the status changed. I had given a detailed response to your queries, but mozilla crashed on me before i clicked on the submit, and i was in a hurry, so i didn't get around to reposting it. in a nutshell: - all browsers tried (opera, netscape & mozilla) - had a URL via e-mail to attend to pending admin requests. - yes it gave the same error with netscape - it was the message received on going to the URL contained in the e-mail. ---------------------------------------------------------------------- Comment By: Marc Merlin (marcmerlin) Date: 2001-07-25 11:14 Message: Logged In: YES user_id=6500 We need the information if you want us to help you ---------------------------------------------------------------------- Comment By: Marc Merlin (marcmerlin) Date: 2001-07-17 08:38 Message: Logged In: YES user_id=6500 What browser? Does it fail repeatetly from different places? Does it fail with plain netscape? Does it fail when you access the page, or when you post back a form? ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=200001&aid=438992&group_id… ----- End forwarded message ----- -- Microsoft is to operating systems & security .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | Finger marc_f(a)merlins.org for PGP key

19 years, 10 months

(2.0.6) pipermail takes >1 minute to rebuild indexes on large lists

by Stephen J. Turnbull

Sorry about the busted references, I'm on the digest. >>>>> "ben" == Ben Gertzfield <che(a)debian.org> writes ben> On a massive list (Mailman 2.0.6) I run that regularly gets a ben> few hundred or more emails every day, things begin to slow ben> down to molasses after a week or two each month, [...] ben> What can I do to help solve this? I know I can use an ben> external archiver, You could switch to weekly or daily archiving, I assume Pipermail supports that. ben> but I don't know anything about that. It's not that hard. I don't know much about this (jason, our Official Mail Guy, has taken an indefinite sanity leave so I'm learning more than I really wanna know), but what XEmacs does is to use the remnants of the former MLM like so: In /etc/aliases: main: "|/home/mailman/prog/wrapper post main",main-archive main-archive: "|/opt/majordomo/wrapper archive2-y2k.pl -a -M -f /opt/archive/majordomo/main "/opt/archive/majordomo/main" is the directory holding the archives. Of course you switch off Mailman's internal archiver. At least this gets the archiving hairball out of Mailman's pipeline. The HTML stuff is done by wilma (Web Interface to List Mail Archives), a front end to mhonarc and glimpse. I don't know anything about that, but could check out the configuration if you want. Provides fairly nice searching, and is admittedly tuned to majordomo archives. Requires one cron job for reindexing. -- University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Institute of Policy and Planning Sciences Tel/fax: +81 (298) 53-5091 _________________ _________________ _________________ _________________ What are those straight lines for? "XEmacs rules."

19 years, 11 months

Re: [Mailman-Users] Upgrading from 2.0b5 to 2.1a2

by barry＠zope.com

[Note: I'm moving this discussion to malman-developers. -baw] >>>>> "JAA" == Jose A Accino <jaccino(a)ieev.uma.es> writes: JAA> Is there any clean way to make a smooth upgrade from 2.0b5 to JAA> lates 2.1a2? (I'm making that just to check the upgrade JAA> process, using a spare machine and a copy of the actual JAA> lists). JAA> Making a 2.1a2 fresh installation (without lists) runs fine. JAA> However, I've been unable to upgrade the old lists. Here are JAA> the main points: There's two ways to upgrade, neither of which has gotten a lot of testing. Based on you're message I tried the first way, and I'm about to check in a bunch of patches to make that go smoother. I'll try the second way soon. 1) Install a fresh virgin MM2.1 in /usr/local/mailman and then move lists from /home/mailman (MM2.0.x) to /usr/local/mailman. This way lets you migrate lists one at a time, gaining confidence in the new system in measured increments. I will probably upgrade {python,zope}.org this way. 2) Install MM2.1 overtop an existing MM2.0.x installation and have everything just magically work. I'll probably upgrade wooz.org's production lists this way. Both ought to work, and ought to be easy. After the checkins, here's a recipe that makes #1 easy: - Do a fresh install of MM2.1 in /usr/local/mailman according to the instructions. - Copy or move `mylist' by first moving /home/mailman/lists/mylist to /usr/local/mailman/lists/mylist and /home/mailman/archives/private/mylist.mbox to /usr/local/mailman/archives/private BTW, I used rsync for this, just cause it's a cool tool and I can actually remember how to invoke it with the right options. :) That's it. Everything else should just work, except for possibly two minor complications. - If your web server is going to support both installations, it's possible you'll have two different base urls, one for your MM2.0.x installation and one for your MM2.1 installation. If so, you'll need to manually change each moved list's web_page_url attribute. Fortunately, I'm about to add a script called `fix_url' which will reset a list's web_page_url to mm_cfg.DEFAULT_URL. So if you configure mm_cfg.py correctly, then run fix_url, you should be golden. I.e. % bin/withlist -l -r fix_url mylist - Same dealie with your MTA, i.e. your wrapper aliases may point to two different directories. Here, because I use Postfix and glued them together as described in README.POSTFIX, all I needed to do was run bin/genaliases and the new list aliases got installed. I'm sure that if you use Exim and glued them together as per README.EXIM, you probably don't need to do anything. (I'm still waiting on contributions for better gluing instructions for Sendmail, qmail, and any others.) -Barry

19 years, 11 months

CPU Usage

by Rodolfo Pilas

Two mails about CPU usage: ----------------------------------------- Date: Tue, 25 Sep 2001 00:50:33 -0300 From: Rodolfo Pilas <rodolfo(a)linux.org.uy> To: mailman-users(a)python.org Subject: [Mailman-Users] CPU Usage in 2.1a2 Hello, Perhaps somebody can explain me why I have a task (mailman) to eat all of my CPU: 60 processes: 56 sleeping, 4 running, 0 zombie, 0 stopped CPU states: 5.1% user, 94.8% system, 0.0% nice, 0.0% idle Mem: 259688K av, 154984K used, 104704K free, 0K shrd, 90484K buff Swap: 385552K av, 12004K used, 373548K free 25160K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 25893 root 14 0 3688 3688 2468 R 99.0 1.4 2:42 python bin/mailmanctl start 25899 root 2 0 924 924 732 R 0.7 0.3 0:01 top Sometimes I have two python task eating 50% of my CPU each one. It is normal? How many time these task are overload the CPU? ---------------------- Date: Mon, 24 Sep 2001 23:57:03 -0400 From: David Ball <dball(a)wcom.ca> To: Rodolfo Pilas <rodolfo(a)linux.org.uy> Subject: Re: [Mailman-Users] CPU Usage in 2.1a2 I have experienced the same problem recently (v2.0.6), and ended up having to disable the Mailman web interface as Python2.1 procs were taking down my machine (a mere P75 w/16MB or ram, which may account for the problem). Unless I killed the processes immediately, all daemons would eventually shut down (sshd, apache, even login), requiring me to reboot the machine when I got home. ------------------------------ -- Rodolfo Pilas Quien los puso a estos tipos donde estan, rodolfo(a)linux.org.uy Quien los deja seguir en su lugar, http://rodolfo.pilas.net Quien los baja ahora de su altar, ICQ #17461636 Quien les paga para que hagan lo que haran http://xtralinux.org -=# Apocalipsis Now % Cuarteto de Nos #=- Public GnuPG key: http://www.keyserver.net 1024D/57153363 2001-06-02 key fingerprint = DAAE 3246 3F7D A420 B7A0 48A5 D120 C773 5715 3363

20 years

Bug in mail->news gateway in 2.0.5 (and *simple* fix)

by "Jürgen A. Erhard"

Bug: in Mailman/Handlers/ToUsenet.py, there are spaces inserted after colons where they are missing ("NNTP is strict about spaces after the colon in headers"). If a colon appears in a continuation line, it is treated as a header colon. Fix: skip lines that start with whitespace. For those interested, the history of the finding of the bug: Problem: my signed emails appeared on Usenet garbled. The multipart boundary had an extra space inserted (which of course the actual boundaries hadn't). After some exchange with Martin Armstrong (who made me aware of this) we concluded it had to be the mail->news gateway (as my direct mail to him, also signed, wasn't garbled). I then investigated ToUsenet.py, and found the error. Why this hasn't hit a lot more often is simple: most MUA seem to use some random gibberish as the boundary string. SEMI doesn't... Of course, if this is fixed in 2.1, you can ignore this (assuming that it comes out soonish ;-) Bye, J -- Jürgen A. Erhard (juergen.erhard(a)gmx.net, jae(a)users.sourceforge.net) MARS: http://members.tripod.com/Juergen_Erhard/mars_index.html Life's Better Without Braces (http://www.python.org) I wish I had more energy -- or less ambition.

20 years

privacy problems with web interface

by Federico Grau

Hello: We're running a 16K member mailing list on version 1.1 and running into a significant privacy issue regarding membership management. As distributed, Mailman makes it trivial to discover if a given address is in fact a subscriber. If you suspect dev(a)null.com has joined a list, go to the user page and enter his address to subscribe; you'll get back a revealing reply 'You already belong, dummy'.. We initially yanked large chunks of html from the general list information page, but that removes all web-based user options. Our ideal outcome is a General List Info Page with two actions: No password section: Current users get passwords (if any) mailed; with the web page not saying if any address is subscribed or not. If not a subscriber, Mailman would send a "subscription confirmation" notice to the potential member. (This also alerts the 'victim' of potential abuse.) Password presented sections: Users w/ password see no feedback until after entering a valid ID and pw; i.e. both non-existent users and existing ones w/bad pw's are both met with "Sorry, wrong password" as in a *nix login. We looked at modifying the html on the user pages but the python module "handle_opts" seems hard-coded into giving revealing responses. We also glanced at Mailman 2.0.6 but it seemed to offer the same behavior. Has anyone else already looked into this issue, and proposed code to solve it? We are considering writing a patch for "handle_opts" and and submitting it but 1) don't want to fork the code, and 2) don't want to duplicate/waste the effort. thanks, David and Federico

20 years

[PATCH] Port HyperArch/pipermail to mimelib

by Ben Gertzfield

Here's a port of HyperArch and pipermail to mimelib. This allows proper parsing of multipart messages, and will make i18n handling much easier. This is a big step forward, I think, because now we no longer have two very different Message classes in Mailman. This patch depends on the mimelib patch I just sent; it uses the get_decoded_payload() function I added to get a nice text representation of even a multi-part message. This will let us even display a message for non-text parts of a message, and eventually will let HyperArch display attachments inline. And of course, as I mentioned in my previous mail, this will prevent base64 gobbeldygook from showing up in the archives. This patch even deals with multiple text/* attachments to a message, and will include them all in the archive even if they're base64 or quoted-printable encoded. It currently does not deal with replacing high-ASCII characters with HTML entities in HyperArch; I'm going to deal with that next by taking the htmlentitydefs module's hash, inverting it, and using that as a big global search-and-replace, if the charset is undefined or iso-8859-1. You can see an example message with lots of mixed parts in my pipermail archive, at: http://nausicaa.interq.or.jp/pipermail/test2/2001-September/000025.html (The original message that produced this output is available at: http://nausicaa.interq.or.jp/~ben/mime-test.txt) . This also patches pythonlib/mailbox.py to use mimelib instead of rfc822. This is the last use of rfc822 in Mailman, so we can now remove pythonlib/rfc822.py completely from the archives -- now we use mimelib entirely! Patch follows, against current Mailman CVS. Index: Mailman/Archiver/HyperArch.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/Archiver/HyperArch.py,v retrieving revision 2.7 diff -u -r2.7 HyperArch.py --- Mailman/Archiver/HyperArch.py 2001/07/26 05:26:48 2.7 +++ Mailman/Archiver/HyperArch.py 2001/09/13 06:19:47 # Note: I'm overriding most, if not all of the pipermail Article class # here -ddm # The Article class encapsulates a single posting. The attributes are: @@ -165,8 +164,8 @@ # Snag the content-* headers. RFC 1521 states that their values are # case insensitive. - ctype = message.getheader('Content-Type', 'text/plain') - cenc = message.getheader('Content-Transfer-Encoding', '') + ctype = message.gettype("text/plain") + cenc = message.get('Content-Transfer-Encoding', '') self.ctype = string.lower(ctype) self.cenc = string.lower(cenc) self.decoded = {} @@ -283,42 +282,14 @@ next = next_wsubj = "" return next, next_wsubj - _rx_quote = re.compile('=([A-F0-9][A-F0-9])') - _rx_softline = re.compile('=[ \t]*$') - def _get_body(self): """Return the message body ready for HTML, decoded if necessary""" try: body = self.html_body except AttributeError: body = self.body - if self.charset is None or self.cenc != "quoted-printable": - return null_to_space(string.join(body, "")) - # the charset is specified and the body is quoted-printable - # first get rid of soft line breaks, then decode literals - lines = [] - rx = self._rx_softline - for line in body: - mo = rx.search(line) - if mo: - i = string.rfind(line, "=") - line = line[:i] - lines.append(line) - buf = string.join(lines, "") - - chunks = [] - offset = 0 - rx = self._rx_quote - while 1: - mo = rx.search(buf, offset) - if not mo: - chunks.append(buf[offset:]) - break - i = mo.start() - chunks.append(buf[offset:i]) - offset = i + 3 - chunks.append(chr(string.atoi(mo.group(1), 16))) - return null_to_space(string.join(chunks, "")) + + return null_to_space(string.join(body, "")) def _add_decoded(self, d): """Add encoded-word keys to HTML output""" Index: Mailman/Archiver/pipermail.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/Archiver/pipermail.py,v retrieving revision 2.8 diff -u -r2.8 pipermail.py --- Mailman/Archiver/pipermail.py 2001/08/17 05:41:25 2.8 +++ Mailman/Archiver/pipermail.py 2001/09/13 06:19:47 @@ -7,6 +7,11 @@ import string import time +from Mailman.pythonlib.StringIO import StringIO + +from mimelib.date import parsedate_tz +from mimelib.address import parseaddr + try: import cPickle pickle = cPickle @@ -20,7 +25,6 @@ from Mailman.Mailbox import Mailbox from Mailman.i18n import _ - msgid_pat = re.compile(r'(<.*>)') def strip_separators(s): @@ -155,7 +159,7 @@ self.parentID = None self.threadKey = None # otherwise the current sequence number is used. - id = strip_separators(message.getheader('Message-Id')) + id = strip_separators(message['Message-Id']) if id == "": self.msgid = str(self.sequence) else: self.msgid = id @@ -169,8 +173,8 @@ self._set_date(message) # Figure out the e-mail address and poster's name - self.author, self.email = message.getaddr('From') - e = message.getheader('Reply-To') + self.author, self.email = parseaddr(message['From']) + e = message['Reply-To'] if e is not None: self.email = e self.email = strip_separators(self.email) @@ -184,11 +188,11 @@ # shouldn't be necessary, but changing this may break code. For # safety, I save the original headers on different attributes for use # in writing the plain text periodic flat files. - self._in_reply_to = message.getheader('in-reply-to') - self._references = message.getheader('references') - self._message_id = message.getheader('message-id') + self._in_reply_to = message['in-reply-to'] + self._references = message['references'] + self._message_id = message['message-id'] - i_r_t = message.getheader('In-Reply-To') + i_r_t = message['In-Reply-To'] if i_r_t is None: self.in_reply_to = '' else: @@ -196,7 +200,7 @@ if match is None: self.in_reply_to = '' else: self.in_reply_to = strip_separators(match.group(1)) - references = message.getheader('References') + references = message['References'] if references is None: self.references = [] else: @@ -209,14 +213,12 @@ if message.has_key(i): self.headers[i] = message[i] - # Read the message body - message.rewindbody() - self.body = message.fp.readlines() + self.body = message.get_decoded_payload() def _set_date(self, message): if message.has_key('Date'): self.datestr = str(message['Date']) - date = message.getdate_tz('Date') + date = parsedate_tz(self.datestr) else: self.datestr = '' date = None Index: Mailman/pythonlib/cgi.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/pythonlib/cgi.py,v retrieving revision 2.0 diff -u -r2.0 cgi.py --- Mailman/pythonlib/cgi.py 2000/12/07 16:53:20 2.0 +++ Mailman/pythonlib/cgi.py 2001/09/13 06:19:47 @@ -30,7 +30,8 @@ import os import urllib import mimetools -import rfc822 +from Mailman import Message +from mimelib import Parser import UserDict from StringIO import StringIO @@ -462,6 +463,7 @@ self.filename = None if pdict.has_key('filename'): self.filename = pdict['filename'] + self.parser = Parser.Parser(Message.Message) # Process content-type header # Index: Mailman/pythonlib/mailbox.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/pythonlib/mailbox.py,v retrieving revision 1.1 diff -u -r1.1 mailbox.py --- Mailman/pythonlib/mailbox.py 2001/02/15 06:09:38 1.1 +++ Mailman/pythonlib/mailbox.py 2001/09/13 06:19:47 @@ -3,13 +3,15 @@ """Classes to handle Unix style, MMDF style, and MH style mailboxes.""" -import rfc822 +from Mailman import Message +from mimelib import Parser import os __all__ = ["UnixMailbox","MmdfMailbox","MHMailbox","Maildir","BabylMailbox"] +parser = Parser.Parser(Message.Message) class _Mailbox: - def __init__(self, fp, factory=rfc822.Message): + def __init__(self, fp, factory=parser.parse): self.fp = fp self.seekp = 0 self.factory = factory @@ -184,7 +186,7 @@ class MHMailbox: - def __init__(self, dirname, factory=rfc822.Message): + def __init__(self, dirname, factory=parser.parse): import re pat = re.compile('^[1-9][0-9]*$') self.dirname = dirname @@ -211,7 +213,7 @@ class Maildir: # Qmail directory mailbox - def __init__(self, dirname, factory=rfc822.Message): + def __init__(self, dirname, factory=parser.parse): self.dirname = dirname self.factory = factory

20 years

(2.0.6) pipermail takes >1 minute to rebuild indexes on large lists

by Ben Gertzfield

On a massive list (Mailman 2.0.6) I run that regularly gets a few hundred or more emails every day, things begin to slow down to molasses after a week or two each month, with the qrunner process taking literally HUNDREDS of megabytes of RAM, and 100% CPU, all the time. It's gotten so bad that the pipermail databases have gotten massive: -rw-rw-r-- 1 list list 40866061 Sep 13 19:24 2001-September-article -rw-rw-r-- 1 list list 686273 Sep 13 19:24 2001-September-author -rw-rw-r-- 1 list list 561479 Sep 13 19:24 2001-September-date -rw-rw-r-- 1 list list 796534 Sep 13 19:24 2001-September-subject -rw-rw-r-- 1 list list 569238 Sep 13 19:24 2001-September-thread And, of course, pipermail takes over a *MINUTE* to rebuild the indexes now. This is with a bunch of syslog() debugging in pipermail.py. Notice the timestamps -- this is how long it takes a *single* message to make it through the ToArchive step of the pipeline.. Sep 13 19:24:02 2001 (29357) in pipermail._update_simple_index() now Sep 13 19:24:02 2001 (29357) opening index as stdout Sep 13 19:24:02 2001 (29357) done opening index, writing index header Sep 13 19:24:02 2001 (29357) getting article 200109010000.JAA26551(a)admin.interq.or.jp from db Sep 13 19:24:03 2001 (29357) writing index entry 0 Sep 13 19:24:03 2001 (29357) going to next entry (snip LOTS of index entry writing -- note the timestamp and the count below) Sep 13 19:24:52 2001 (29357) getting article 3B95AB3EA.4F08HONDA(a)192.168.1.190 from db Sep 13 19:24:52 2001 (29357) writing index entry 4794 Sep 13 19:24:52 2001 (29357) going to next db Sep 13 19:24:52 2001 (29357) calling self.write_index_footer() Sep 13 19:24:52 2001 (29357) in pipermail._update_thread_index Sep 13 19:24:53 2001 (29357) in pipermail.write_TOC() Sep 13 19:24:53 2001 (29357) done writing TOC Sep 13 19:24:53 2001 (29357) picking archive state into /var/lib/mailman/archives/private/sysadmin/pipermail.pck While it does this, the size of the python process that's doing the qrunner grows by *megabytes* each second. I've seen it get to 200MB before it finishes, and it's always taking up 99% CPU. As you can guess, things rapidly get out of hand, and messages start arriving faster than pipermail can deal with them. And since it's first in the pipeline, nothing gets delivered until pipermail archives the mail.. and mail queues up and queues up, until it's taking 7-8 hours to send out each mail -- no joke. What can I do to help solve this? I know I can use an external archiver, but I don't know anything about that. I've commented out the ToArchive step from the pipeline for now, but I don't know what else I can do. There *MUST* be a way to not have to loop over every single article in the simple index every time a message arrives, right? Other people must have run into this, and I'd like to know what I can do to only have to write the changed articles in the simple index. Thanks, Ben -- Brought to you by the letters F and O and the number 12. "Johnny! Don't go! It's too dangerous!" "I don't care!" Debian GNU/Linux maintainer of Gimp and GTK+ -- http://www.debian.org/

20 years

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

Mailman-Developers September 2001