Mailman 3 May 2003 - Mailman-Developers

message data storage
by Donn Cave May 7, 2003

May 7, 2003

I'm breathing easier this week, now that a new month has started. At the end of April, one of our new lists was up to 6500 archived messages for the month. It takes close to 4 minutes CPU time to archive a message in a situation like that. The average delivery rate to achieve 6500 messages in 30 days is 1 per 6 minutes, so the system obviously cannot work. I intervened and threw away a couple hundred of them (it's drivel) to break the logjam and get everyone else going. We have only a fraction of our expected population so far, we're just beginning to migrate in earnest from listproc, so it looks like we're in some trouble. Most of the work appears to be in the 68Mb database, as in database/2003-April-article. It can take up to a minute to get this through marshal.load(), almost all of it disk I/O. It takes nearly 3 minutes CPU time to get all the values through cPickle.loads(), and I believe the archiver does in fact load every single one. Working with the current system (as opposed to the Next Generation archiver), I'm tempted to think about a semi-permanent per-archive archiver and queue. The archiver would exit when its queue has been empty for some short interval, but any extra messages that arrive during that time get the database for free. I'd be interested in any other ideas. thanks, Donn Cave, donn(a)u.washington.edu

3 2

re: advice needed: shunt problems with MM2.1 and 2.1.2 only on one list out of many
by Chris Dagdigian May 7, 2003

May 7, 2003

Hello, I'm following up my own message because we seem to have resolved our issue with MM2.1.2 shunting messages with a "TypeError: iterable argument required" traceback. We re-created our list and got email flowing again which was our main concern. Andrew Dalke then had the time to check out the traceback errors and the messages that remaind in the shunt/ directory. He is suggesting a patch or a possible fix so I'm enclosing his entire message here: Our server is an ancient Alpha box running RedHat Linux 6.2 with a clean build of python 2.3. Mailman is 2.1.2 but since we've been using MM for years and upgrade-installing into the same directory I can't promise that our problems were not caused entirely by old cruft laying around.... -Chris Here are the notes from Andrew: > Actually, I didn't get your email about the shunt problem, I got it > about the archive problem. So I'm sending this to you to let > you forward to mailman developers in the proper context > > Here's the traceback > > Traceback (most recent call last): > File "/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop > self._onefile(msg, msgdata) > File "/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile > keepqueued = self._dispose(mlist, msg, msgdata) > File "/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose > more = self._dopipeline(mlist, msg, msgdata, pipeline) > File "/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline > sys.modules[modname].process(mlist, msg, msgdata) > File "/home/mailman/Mailman/Handlers/ToDigest.py", line 89, in process > send_digests(mlist, mboxfp) > File "/home/mailman/Mailman/Handlers/ToDigest.py", line 130, in send_digests > send_i18n_digests(mlist, mboxfp) > File "/home/mailman/Mailman/Handlers/ToDigest.py", line 303, in send_i18n_digests > msg = scrubber(mlist, msg) > File "/home/mailman/Mailman/Handlers/Scrubber.py", line 257, in process > url = save_attachment(mlist, part, dir) > File "/home/mailman/Mailman/Handlers/Scrubber.py", line 349, in save_attachment > ext = guess_extension(ctype, fnext) > File "/home/mailman/Mailman/Handlers/Scrubber.py", line 80, in guess_extension > if ext in all: > TypeError: iterable argument required > > It occurs when Scrubber.guess_extension . The code does > > all = guess_all_extensions(ctype, strict=0) > if ext in all: > return ext > return all and all[0] > > The function 'guess_all_extensions' comes from the top of the > files from the section > > try: > from mimetypes import guess_all_extensions > except ImportError: > import mimetypes > def guess_all_extensions(ctype, strict=1): > # BAW: sigh, guess_all_extensions() is new in Python 2.3 > > We're running Python 2.3, which means we're using the > one from mimetypes. That function's docstring says > > guess_type(). If no extension can be guessed for `type', None > is returned. > > So if no extension can be guessed, then Scrubber.guess_extension > try to iterate over None, which fails > > Our fix to get things working again is > > all = guess_all_extensions(ctype, strict=0) > if all is None: all = [] > if ext in all: > return ext > return all and all[0] > > However, another possibility is that the guess_all_extensions > in Python 2.3's mimetypes should return [] when it cannot > guess, rather than a None. I would be happier with that > solution than this patch, but will leave it up to the Mailman > and Python people to decide on the appropriate solution. > > Andrew Dalke > dalke(a)dalkescientific.com

1 0

Re: [Mailman-Developers] message data storage
by Donn Cave May 6, 2003

May 6, 2003

Quoth Simone Piunno <pioppo(a)ferrara.linux.it>: | On Tuesday 06 May 2003 23:39, Donn Cave wrote: | | > At the end of April, one of our new lists was up to 6500 archived | > messages for the month. It takes close to 4 minutes CPU time to | > archive a message in a situation like that. The average delivery | > rate to achieve 6500 messages in 30 days is 1 per 6 minutes, so | > the system obviously cannot work. | | Why don't you just change the archiving period to "weekly"? Good question. Maybe because I spend to much time rewriting Mailman and not enough time using it. But I'm not sure that's really solving the problem, though it would alleviate it in this case. If I understand the archive volume thing, it's a user-visible change. We could have a talk with the list owner, and he'd probably be happy to do this - or for that matter, would probably be happy to dispense with archives altogether, which would be even better! But we have several thousand lists. 1500 already in Mailman, 3 times that many yet to be converted from listproc. The present problem is new to me partly because it's an unusually active list, but also because up to now the relatively light load on our host has allowed this kind of problem to pass unnoticed. This summer when we have twice the population, will we have only twice as many problem lists - or will we have 20 times as many, because the higher overall load has lowered the threshhold? Do we want to try to manage this problem, or solve it? I'm hoping for something more like a solution. thanks, Donn Cave, donn(a)u.washington.edu

1 0

advice needed: shunt problems with MM2.1 and 2.1.2 only on one list out of many
by Chris Dagdigian May 6, 2003

May 6, 2003

Hello, {First off thanks for a fantastic package...we've been using MM for several years on bioperl.org} I've got one list out of many on our MM server that has not processed a new email since May 1st. The messages are apparently all in the shunt/ dir and running bin/unshunt just repeats the same traceback errors. Tried googling, tried the archives and tried upgrading from 2.1 to 2.1.2 but I'm still having a serious problem with the bioperl-l(a)bioperl.org mailing list. It also happens to be our biggest and most active list so I'm starting to get a bit stressed. Google searching on "shunt errors" and "TypeError: iterable argument required" brought me to a page where it was suggested that there was a unicode parsing bug that triggers when subscribers have 8-bit encoded chars. I did find one subscriber who had funky chars in his "name" field and deleted it via the web admin. Running "list_list" verified that the subscriber list looks OK otherwise. Sent a test message to the list and it still got shunted with a traceback error message. The message.pck file is attached. I'll happily supply the .db file if it would be useful. The actual traceback is this: > May 06 16:31:39 2003 (14914) Uncaught runner exception: iterable argument required > May 06 16:31:39 2003 (14914) Traceback (most recent call last): > File "/home/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop > self._onefile(msg, msgdata) > File "/home/mailman/Mailman/Queue/Runner.py", line 155, in _onefile > keepqueued = self._dispose(mlist, msg, msgdata) > File "/home/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose > more = self._dopipeline(mlist, msg, msgdata, pipeline) > File "/home/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline > sys.modules[modname].process(mlist, msg, msgdata) > File "/home/mailman/Mailman/Handlers/ToDigest.py", line 89, in process > send_digests(mlist, mboxfp) > File "/home/mailman/Mailman/Handlers/ToDigest.py", line 130, in send_digests > send_i18n_digests(mlist, mboxfp) > File "/home/mailman/Mailman/Handlers/ToDigest.py", line 303, in send_i18n_digests > msg = scrubber(mlist, msg) > File "/home/mailman/Mailman/Handlers/Scrubber.py", line 257, in process > url = save_attachment(mlist, part, dir) > File "/home/mailman/Mailman/Handlers/Scrubber.py", line 349, in save_attachment > ext = guess_extension(ctype, fnext) > File "/home/mailman/Mailman/Handlers/Scrubber.py", line 80, in guess_extension > if ext in all: > TypeError: iterable argument required > > May 06 16:31:39 2003 (14914) SHUNTING: 1052253095.709591+38e71a4dd22ea87b3961d1aa1c14d08e16d2683c Any advice or pointers would be deeply appreciated. Regards, Chris open-bio.org list admin -- Chris Dagdigian, <dag(a)sonsorol.org> BioTeam Inc. - Independent Bio-IT & Informatics consulting Office: 617-666-6454, Mobile: 617-877-5498, Fax: 425-699-0193 PGP KeyID: 83D4310E Yahoo IM: craffi Web: http://bioteam.net

1 0

English (USA)
by Paul Traina May 6, 2003

May 6, 2003

Not to beat a dead horse, but ... > So far, I've only noticed one, the spelling of "internationalisation", and that only appears on an admin page. Look at a British English dictionary and you'll note that the 'ize' form is actually primary, although not common usage. Can't we just all get along? It seems ridiculous to have an en_US and en_GB for such triviality. Anything that differs can probably be rewritten to be proper in both dialects. Yours for "English" as opposed to "English (...)"

1 0

Converting from MM1.1 to MM2.1.1
by Grant Cullen May 5, 2003

May 5, 2003

Yes I am making that big a jump. MM1.1 is running currently, we are replacing the server and in the process upgrading all of our software. The problem is: New lists created on the new system show up when the listinfo page is displayed. Lists which are copied over from the old system do no appear on the listinfo page, but are accessible. Obviously the configuration files are different, but mailman seems to be able to handle that. Running the config_list command on new list and a copied over list produce similar output. Suggestions would be appreciated. -- Grant Cullen <grant(a)libertyrms.info> Liberty RMS

1 0

Uncaught bounce ... still in mailman 2.1.2
by Martin Maechler May 5, 2003

May 5, 2003

Dear developers here's an example of an "uncaught bounce notification" where I believe mailman *should* catch. Regards, Martin

1 0

Members' documentation for 2.1
by Terri Oda May 5, 2003

May 5, 2003

I've checked in member (subscriber, user) documentation for 2.1. It's available in cvs or on my website. CVS gives only the tex source, and since I imagine people would rather read the final output, I've also uploaded some generated copies to my website in various formats. Some proofreading's been done by people who are list users (but aren't list maintainers), but I'm sure more comments would make for a better document, so feel free to make suggestions or point out missed typos! Link for the web cvs: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mailman/mailman/doc/mailman-… Link for the generated copies on my website: http://terri.zone12.com/doc/ Terri

2 1

2.1.2 -- still "Uncaught runner exception: unknown encoding"
by Martin Maechler May 5, 2003

May 5, 2003

Dear Barry and co-developers, thanks a lot for 2.1.2 and all the bugs you've fixed! With 2.1.1 (installed early March here), I had started to allow for some mailing lists all available different languages, including Japanese, e.g. and I see that some users gladly chose them. Maybe that's the reason for things like the following {cut&paste}ed from the data/logs/error file Regards, Martin Maechler <maechler(a)stat.math.ethz.ch> http://stat.ethz.ch/~maechler/ Seminar fuer Statistik, ETH-Zentrum LEO C16 Leonhardstr. 27 ETH (Federal Inst. Technology) 8092 Zurich SWITZERLAND phone: x-41-1-632-3408 fax: ...-1228 <>< -------- cut from data/logs/error ----------------------------- May 05 09:52:37 2003 (23520) SHUNTING: 1052121156.282855+3ec8467f1fdc497d2189844244854948fc4c7c6b May 05 09:52:40 2003 (23520) Uncaught runner exception: unknown encoding May 05 09:52:40 2003 (23520) Traceback (most recent call last): File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/CommandRunner.py", line 218, in _dispose res = Results(mlist, msg, msgdata) File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/CommandRunner.py", line 75, in __init__ subj = make_header(decode_header(subj)).__unicode__() File "/usr/local/app/mailman-sfs/2.1.2/pythonlib/email/Header.py", line 144, in make_header h.append(s, charset) File "/usr/local/app/mailman-sfs/2.1.2/pythonlib/email/Header.py", line 272, in append ustr = unicode(s, incodec, errors) LookupError: unknown encoding May 05 09:52:40 2003 (23520) SHUNTING: 1052121160.428594+dd539dae2deaafc3ecd67c30e303dfaac826f882 May 05 09:52:47 2003 (23520) Uncaught runner exception: unknown encoding May 05 09:52:47 2003 (23520) Traceback (most recent call last): File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/CommandRunner.py", line 218, in _dispose res = Results(mlist, msg, msgdata) File "/usr/local/app/mailman-sfs/2.1.2/Mailman/Queue/CommandRunner.py", line 75, in __init__ subj = make_header(decode_header(subj)).__unicode__() File "/usr/local/app/mailman-sfs/2.1.2/pythonlib/email/Header.py", line 144, in make_header h.append(s, charset) File "/usr/local/app/mailman-sfs/2.1.2/pythonlib/email/Header.py", line 272, in append ustr = unicode(s, incodec, errors) LookupError: unknown encoding May 05 09:52:47 2003 (23520) SHUNTING: 1052121167.541289+1866a13cdc22d889275125bb124ecb58ae5fceb9

1 0

public and private archives problem
by Kory Wheatley May 5, 2003

May 5, 2003

I'm getting this below error when an user tries to open up there archives. I've run check_perms and all the permissions are correct. When I change the archive options on this list to public I get this below error when I change it back to private I can view the threaded archives fine, is there a solution to this. I've tested this with my other lists and it does the same thing. There most be something that I've got configured wrong that a list can't be setup to view the archives in the (public) setting. ACCESS DENIED ! You don't have permission to access the requested directory. There is either no index document or the directory is read-protected. If you think this is a server error, please contact the webmaster ERROR 403 mm.isu.edu Wed Apr 30 11:47:13 2003 Apache/2.0.40 (Red Hat Linux) -- Kory Wheatley Academic Computing Analyst Sr. Phone 282-3874 ######################################### Everything must point to him.

4 4