Mailman 3 October 2003 - Mailman-Developers

ImportError: No module named korean revisted
by John Oct. 25, 2003

Oct. 25, 2003

This probably is a mailman-users issue...but Redhat calls the python v2.2 executable python2 so you need the configure parameter like this make clean ./configure --with-python=/usr/bin/python2 make install

1 0

Memory resource issue with new 2.1.3 BounceRunner
by John Oct. 25, 2003

Oct. 25, 2003

While testing 2.1.3 on a recent mailing I discovered that the BounceRunner process had grown to 400 megabytes! This recent mailing generated a very large number of bounces due to a problem I was having with aol.com The new BounceRunner pre-processes all bounces by building an in memory dictionary with the bounce information. Unfortunately there are no limits placed on the number of bounces that are added to this dictionary. In this case, I received over 10,000 bounces which required over 400 megabytes of RAM to hold. Needless to say my machine slowed to a crawl and continued to add to the structure since it couldn't empty the qfiles/bounces directory fast enough. I think we need a limit on the number of bounces added to this in memory structure regardless of the number of bounces in qfiles/bounces. John Co webmaster momsview.com

1 0

oh, sigh.. It's back.
by Chuq Von Rospach Oct. 24, 2003

Oct. 24, 2003

Hey, barry? remember this? <http://mail.python.org/pipermail/mailman-developers/2003-May/ 015157.html> well, I just did a CVS update to take my system to 2.1.3, and... It's back. make[1]: Nothing to be done for `all'. /usr/bin/python ../build/bin/msgfmt.py -o cs/LC_MESSAGES/mailman.mo cs/LC_MESSAGES/mailman.po Merging new template file with existing translations msgmerge -U da/LC_MESSAGES/mailman.po mailman.pot msgmerge: invalid option -- U Try `msgmerge --help' for more information. make[1]: *** [da/LC_MESSAGES/mailman.po] Error 1 make[1]: Nothing to be done for `all'. 7.050u 4.120s 0:11.85 94.2% 0+0k 0+27io 0pf+0w this is after: 232 21:19 cvs -q up -d -P -A 239 21:32 make distclean 240 21:32 ./configure --with-mail-gid=daemon 241 21:33 make

2 1

Bounce removal parameters default values
by Greg Stark Oct. 23, 2003

Oct. 23, 2003

I find I am being removed from mailman mailing lists left and right. I believe the default values for the bounce removal should be reconsidered. It's possible that you haven't had many users in my situation and so haven't really had a chance to tune these parameters on the low end yet. But they clearly aren't working for me at a few sites. My particular situation is that my site has seen fit to filter viruses by refusing delivery. This causes a bounce from the remote MTA every time someone sends me an Outlook virus. Why my site administrators felt this was necessary is a question for another day, it's not like I use Outlook or like my spam filters wouldn't have thrown these messages away anyways, but whatever. The net result is that some small fraction of messages to me bounce and list management software notices this. The only reason I became aware of the problem was because ezmlm also does this type of processing but it sends a warning message before removing users. It only removes you if the warning message itself bounces. In fact it sends two such warning messages and only removes the user if *both* bounce. This provides the user with a chance to react to the first message and fix the problem -- if they ever see the message. I could beg for a similar feature in mailman, but I'm not sure it's necessary. But I am sure it's necessary to tune the bounce processing parameters. The relatively few bounces I'm generating shouldn't be causing me to get removed when all the real messages are being delivered fine. It seems the legitimate messages that are correctly delivered should reset the count of bounces to 0. Reading the source it seems it has to see DEFAULT_MAX_POSTS_BETWEEN_BOUNCES such legitimate posts between messages. I'm fairly convinced this parameter should always be 0. If any successful delivery occurs the user should never be removed due to bounces. What I don't understand is how DEFAULT_MAX_POSTS_BETWEEN_BOUNCES relates to the parameters I see in the admin. None of the parameters in the admin corresponds to this. How is it calculated? -- greg

6 15

ImportError: No module named korean revisted
by Bob Tanner Oct. 22, 2003

Oct. 22, 2003

% cat /etc/redhat-release Red Hat Linux release 7.3 (Valhalla) $ rpm -qa | grep python python-popt-0.8.8-7.x.2 ---->python2-devel-2.2.2-11.7.3 python-clap-1.0.0-3 python-xmlrpc-1.5.1-7.x.3 rpm-python-4.0.4-7x.18 python-devel-1.5.2-43.73 python-1.5.2-43.73 ---->python2-2.2.2-11.7.3 $ pwd /usr/src/redhat/BUILD/mailman-2.1.3 $ make install Compiling /var/tmp/ZZZZ/Mailman/versions.py ... Traceback (most recent call last): File "bin/update", line 45, in ? import paths File "bin/paths.py", line 59, in ? import korean ImportError: No module named korean make: *** [update] Error 1 I followed the thread here: http://mail.python.org/pipermail/mailman-users/2002-December/024814.html It looks like this issues was "fixed", but I'm still getting it on a RH73 build. I have -devel packages installed. Do I need a "newer" version of python? -- Bob Tanner <tanner(a)mn-linux.org> | Phone : (952)943-8700 http://www.mn-linux.org, Minnesota, Linux | Fax : (952)943-8500 Key fingerprint = AB15 0BDF BCDE 4369 5B42 1973 7CF1 A709 2CC1 B288

1 0

qmail VERP patch changes
by Andrew A. Raines Oct. 22, 2003

Oct. 22, 2003

I'm not sure if Colin still reads this list, but I had to make a couple of changes to the qmail VERP DELIVERY_MODULE on Sourceforge[1]. Namely, the default QVERP_FORMAT didn't make sense, but a docstring addition seemed warranted as well. A patch to Qmail.py is attached. -Drew Footnotes: [1] https://sourceforge.net/tracker/?func=detail&atid=300103&aid=645513&group_i…

2 1

Help with bugs
by Vitaliy Karlov Oct. 22, 2003

Oct. 22, 2003

Good day all. Some days ago I notice than some post to list did not go to list, but stay shuting. in logs/error: this === Oct 22 16:44:21 2003 (38679) SHUNTING: 1066822120.244683+5ac80f2a2c5793309cfbcbed703615ccf31b365b Oct 22 16:44:21 2003 (38679) Uncaught runner exception: 'ascii' codec can't decode byte 0xce in position 1: ordinal not in range(128) Oct 22 16:44:21 2003 (38679) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 110, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 160, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose mlist.ArchiveMail(msg) File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 208, in ArchiveMail h.processUnixMailbox(f) File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 560, in processUnixMailbox self.add_article(a) File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 592, in add_article temp = self.format_article(article) File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 1225, in format_article self.__processbody_URLquote(lines) File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 1164, in __processbody_URLquote text = re.sub('@', _(' at '), text) File "/usr/local/lib/python2.3/sre.py", line 143, in sub return _compile(pattern, 0).sub(repl, string, count) UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 1: ordinal not in range(128) === or this === Oct 22 16:44:20 2003 (38682) SHUNTING: 1066819196.75235+30e4de05fe9d8ee5948c26cd52616945c651f4cb Oct 22 16:44:20 2003 (38682) Uncaught runner exception: Oct 22 16:44:20 2003 (38682) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 110, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 160, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/CookHeaders.py", line 75, in process prefix_subject(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/CookHeaders.py", line 238, in prefix_subject headerbits = decode_header(subject) File "/usr/local/mailman/pythonlib/email/Header.py", line 113, in decode_header raise HeaderParseError HeaderParseError == I thanks for any help. My knowledge in language python is poor. -- WBR, Vitaliy Karlov [KV1670-RIPE]

1 0

Bug in mailman 2.1.3
by Jon Scott Stevens Oct. 22, 2003

Oct. 22, 2003

Hello everyone. I just got the stack trace below from one of my list admins...not sure how he screwed things up...but I figured I would let you know...based on the traceback, it seems that he somehow got a strange character into the list of members... [share] 3:16pm ~ > ./bin/check_db -v sfindie-fence List: sfindie-fence /usr/local/mailman/lists/sfindie-fence/config.pck: okay /usr/local/mailman/lists/sfindie-fence/config.pck.last: okay /usr/local/mailman/lists/sfindie-fence/config.db: okay /usr/local/mailman/lists/sfindie-fence/config.db.last: okay [share] 3:17pm ~ > ./bin/list_members -u sfindie-fence [no output] [share] 3:18pm ~ > ./bin/list_members -i sfindie-fence Êknights_bishop(a)yahoo.com [share] 3:18pm ~ > ./bin/list_members -i sfindie-fence > bad.txt [share] 3:19pm ~ > ./bin/remove_members -f bad.txt sfindie-fence Yup...removing that address solved the problem. Note: this list admin only has access to the web interface. So, somehow he was able to send corrupt data to the server which caused this problem. Probably not a good thing. =) --------------------------------------------------------------------------- Bug in Mailman version 2.1.3 We're sorry, we hit a bug! If you would like to help us identify the problem, please email a copy of this page to the webmaster for this site with a description of what happened. Thanks! Traceback: Traceback (most recent call last): File "/usr/local/mailman/scripts/driver", line 87, in run_main main() File "/usr/local/mailman/Mailman/Cgi/admin.py", line 192, in main show_results(mlist, doc, category, subcat, cgidata) File "/usr/local/mailman/Mailman/Cgi/admin.py", line 491, in show_results form.AddItem(membership_options(mlist, subcat, cgidata, doc, form)) File "/usr/local/mailman/Mailman/Cgi/admin.py", line 799, in membership_options all = [_m.encode() for _m in mlist.getMembers()] UnicodeError: ASCII decoding error: ordinal not in range(128)

2 1

Archive list by date without the date field
by Luigi Rosa Oct. 22, 2003

Oct. 22, 2003

Hello, selecting the date sort in the archive list, the message list that appears does not show the date field. If I am searching a message with a known date, I have to open some messages to "guess" the date. I think that the archive message list should contain the date, at least if the user selects to sort it by date. -- Best regards, Luigi

1 0

Bounce processing observations 2.1.3 versus 2.1.1 for large lists
by John Oct. 22, 2003

Oct. 22, 2003

Greetings: I currently manage a mailman installation on a Redhat system with a total of 229,000 subs spread fairly evenly over 72 announce only lists (averaging 3k users each) . Recently I upgraded from 2.1.1 to 2.1.3 primarily because of the fix for the cross site scripting bug but also for the bounce processing improvements. Prior to the update I would only run BounceRunner every 8 hours because of the large CPU and I/O load it would put on my system (90% CPU, and LOTS of disk I/O). So far it appears that the 2.1.3 bounce processing software is MUCH faster. In many cases it's able to process up to 15 bounces per SECOND. Fantastic. That is versus 6 bounces per second on 2.1.1 (This is a 2GHZ P4 with ATA100 IDE drives) However, with this increased performance (no doubt due to the fact that BounceRunner registers MANY bounces for one list at time) comes a problem best illustrated by the following excerpt from my bounce log: Oct 20 12:29:54 2003 (3706) Processing 1211 queued bounces Oct 20 12:33:04 2003 (3706) bouncingsubscriber1(a)isp.com: momsviewf current bounce score: 5.0 ..... dozens and dozens more entries for the momsviewf list ..... Oct 20 12:36:10 2003 (3706) bouncingsubscriber2(a)isp2.com: momsviewf current bounce score: 2.0 If I understand how the processing is done correctly, was the momsviewf list indeed locked for a period of 3 minutes and 6 seconds? I noticed also that I could not access the momsviewf list via the web admin interface, it would hang on the admin password entry screen. The momsviewf list has 4374 subs. I did not notice this issue on 2.1.1. Probably since it releases the list lock after every bounce. While increasing the performance of bounce processing significantly, the changes in 2.1.3 appear to have created a lock contention issue as a side effect. My suggested fix, as I previously mentioned in my Jan 30, 2003 posting to this list reproduced below is to purposely LIMIT the number of bounces processed (preferably per list) so that the lock is released in a reasonable time period. I would be interested in hearing other suggestions or observations about this issue. Thanks John Co-webmaster momsview.com ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- intialize x to number of bounces to process on each pass While Forever Initialize Python list structure to hold bounces (Process x emails in the bounce queue) For x emails in queue Dequeue the message Extract addresses to bounce SAVE address and Listname in Python list structure If Python List structure contains emails For all mailing lists in Python structure REREAD list from disk LOCK the LIST For all addresses that bounced for this list Register Bounce SAVE the list to disk UNLOCK the list SLEEP for SLEEPTIME CLEANUP on exit Advantages to this method: (1) We process a number of bounces before writing out the list reducing I/O (the real bootleneck) by factor x. When x is one the algorithm almost degenerates to the current method (2) Since we always sleep on each pass it gives other processes (like the Web gui) a chance to read the list. (3) By increasing x we control the number of bounces that get processed on each pass. The time it takes to extract the addresses gives other processes time to acquire the list lock and avoid "lockout" (4) Since "in memory" bounce registration is very fast we can do a lot of them while the list is locked without adding significantly to the already long lock time on a big list (I believe the I/O is the limiting factor)

1 0