Mailman 3 May 2003 - Mailman-Developers

Mailman 2.1.2 - Logging error messages during start and stop
by Oliver Egginger 16 May '03

16 May '03

Hello, I run Mailman 2.1.2. I can create lists, subscribe / unsubscribe and send messages to my lists. Also everything else (which I have tested) seems to work fine. But in verbose mode I receive during the startup (mailmanctl start) and during a mailmanctl stop messages like the following: # clear __builtin__._ Logging error: <StampedLogger to '/home/mailman/logs/error'> Traceback (most recent call last): File "/home/mailman/Mailman/Logging/MultiLogger.py", line 43, in write logger.write(msg) File "/home/mailman/Mailman/Logging/StampedLogger.py", line 73, in write Logger.write(self, "%s %s" % (prefix, msg)) File "/home/mailman/Mailman/Logging/Logger.py", line 88, in write msg = unicode(msg, self.__encoding) SystemError: codec module not properly initialized Original log message: # clear __builtin__._ Only the lines 43, 73 and 88 of MultiLogger.py are affected, nothing else. Augur this "Loggin errors" something bad? The loggin seems to work well anyway. I receive this messages only in verbose mode. There are no straying entries in /home/mailman/locks. I wondering, if I should bring this mailman installation into production? - oliver

1 0

Re: Looking at performance again
by Charlie Clark 15 May '03

15 May '03

On 2003-05-14 at 21:11:38 [+0200], mailman-developers-request(a)python.org wrote: > > I'm not sure I understand the multiple SMTP backends. MAL says "for > delivery", which I take to mean incoming. Our listproc setup delivers to > a single host but sends from multiple hosts - that's time consuming work > that can easily be parallelized. We'll do it with Mailman too, have just > been procrastinating. The SMTP inbound traffic seems relatively trivial > to me. No, it is outbound "delivery of payload" - our application is large volume outgoing with negligible incoming. Using delivery makes sense for us in the overall sense and we use multiple SMTPs to make this as efficient as possible. Charlie

2 2

Re: A Helper Script
by Mark Rauterkus 14 May '03

14 May '03

> Subject: [Mailman-Developers] A Helper Script ... > Here's a link to the script: > http://linux.ucla.edu/~larva/projects/list_handle_pending Wow. Great script to have. If it works, I hope it becomes part of the /bin utilities. Having a handto get out of hot water is always nice. Having that hand always available and at the ready is even nicer. Ta. Mark Rauterkus mark(a)Rauterkus.com http://Rauterkus.com

3 2

Another Traceback
by Mike Burton 14 May '03

14 May '03

Barry got me steered in the right direction on my last problem (mixed python versions on this machine), and I appreciate that a bunch. All seemed to be well until this morning when I attempted to unsubscribe somebody from one of the lists. I tested a bit further and I get the same traceback. Again, any clues will be greatly appreciated. I'm not great at python - sorry. Traceback (most recent call last): File "/usr/local/mailman/scripts/driver", line 87, in run_main main() File "/usr/local/mailman/Mailman/Cgi/admin.py", line 169, in main change_options(mlist, category, subcat, cgidata, doc) File "/usr/local/mailman/Mailman/Cgi/admin.py", line 1354, in change_options mlist.ApprovedDeleteMember(user) File "/usr/local/mailman/Mailman/MailList.py", line 944, in ApprovedDeleteMember self.SendUnsubscribeAck(emailaddr, userlang) File "/usr/local/mailman/Mailman/Deliverer.py", line 87, in SendUnsubscribeAck Utils.wrap(self.goodbye_msg), lang) File "/usr/local/mailman/Mailman/Message.py", line 206, in __init__ errors='replace') TypeError: __init__() got an unexpected keyword argument 'errors' Take care, Mike

1 0

Re: [Mailman-Developers] Looking at performance again
by Donn Cave 14 May '03

14 May '03

Our disk I/O problems will come up towards the end of the month, with database/2003-May-article 2x the size of 2003-May.txt. Don't know if this counts in raw throughput, but it sure has an impact. Archiving looks to me like the bulk of our system CPU and disk load. Even less in the raw throughput department, we generate a static main listinfo page rather than poke through thousands of config.pcks every time someone wants to see the page. - In principle threading looks like a can of worms to me. Still incomplete support across platforms, problems with things like signal delivery, increase in complexity and decrease in robustness. It does depend on the application, but you have to be desperate. - We don't use NFS, but it's easy to see why someone would want to - run the web interface and mailman on separate hosts, etc. - I'm not sure I understand the multiple SMTP backends. MAL says "for delivery", which I take to mean incoming. Our listproc setup delivers to a single host but sends from multiple hosts - that's time consuming work that can easily be parallelized. We'll do it with Mailman too, have just been procrastinating. The SMTP inbound traffic seems relatively trivial to me. Donn Cave, donn(a)u.washington.edu

2 1

Looking at performance again
by Barry Warsaw 14 May '03

14 May '03

I've been doing some (limited) performance testing lately, and I wanted to share some numbers and get some feedback. I've also been having fun re-reading some old mm-dev threads related to performance. :) I'm specifically looking for places to improve Mailman's raw throughput. I understand that MTA tuning can have a huge impact on the system, but I think that subject's been hashed out quite well in the past. On the table are anything from low-hanging fruit hacks to mm3-level redesigns. What I actually can implement all depends. :) I've been testing the following set up: - Postfix 2.0.9 configured with a special test transport such that all email @example.com gets dd'd to /dev/null - Postfix running on the same machine as Mailman 2.1.2+ and a second test with Postfix (similarly configured) on a separate, very unloaded, but less beefy machine sitting next to me on a 100Mb ethernet. - RH9 2.4.20-9 kernel, 863Mhz Dell PIII, 512MB (1723 bogomips), ext3, a WDC IDE drive of some 2 y.o. vintage. - Python 2.2.2 built from source My list consists of 8000 members like abcdefg(a)example.com where the localpart varies randomly. I've tried deliveries of 10KB, 50KB, 220KB, 1MB of text/plain and a 220KB multipart/related snapshot of a web page [1]. I have VERP and personalization both turned on. I started looking at memory usage, but I'm not so concerned about that now. It may be something to address later but I think it's "reasonable". First the (approximate) numbers. All deliveries are to 8000 members, each with their own personalized copy. SMTP_MAX_RCPTS is 500 unless otherwise specified (minimal impact seemingly). msgsz type time msg/hour ----- ---- ---- -------- 10k plain/text 6min 80k/hr 50k p/t 9.5min 50k/hr (SMTP_MAX_RCPTS=5) 220k p/t 24m 20k/hr 1MB p/t 105m 4500/hr 220k m/related 44m 10k/hr 220k m/related 41m 11k/hr (SMTP_MAX_RCPTS=5) 220k m/related 46m 10k/hr (remote MTA) A few high-level bits: - Disk i/o probably isn't much of an issue. Once the message hits the out qrunner, it's only two files and all the personalization weaving happens in memory just before the message goes out on the socket. Since using a remote MTA was actually slightly slower, I'm guessing that MTA overhead in the /dev/null pipe is actually minimal (the remote machine is a 500MHz, 128 KB, 999 bogomips, mostly idle). - email.Generator.Generator (and email.Parser.Parser) are good candidates for optimization. You can see that with the 220KB messages, the fact that one has structure and the other doesn't, affects performance significantly. That doesn't surprise me. ;) - Even so, a factor of 100 in message size has a 20x hit on performance. Part of that may be the way the personalization weaving gets done. Right now, we make a copy.deepcopy() of the original message object model, then poke in the personalization parts in the headers and such, then do all the complex stuff in Decorate.py (footers, headers, etc.), then generate the flat text. Maybe we can speed things up by converting the message to flat text as early as possible and just doing string substitution at the point of weaving. - What kind of a hit does the memberdb-in-a-pickle take? Would things go faster if we stored the member data in a Berkeley, MySQL, or other real database? I'd like to do some testing with my BDB member code and I'm wondering if the folks working on other member adapters have any performance feedback. - XVERP might be interesting, but it seems useless for personalization. - Do we win or lose with the process model, as compared to say, a threading model? I've been wondering if our fears of the Python GIL are unfounded. We could certainly reduce memory overhead by multi-threading, and we might be able to leverage something like Twisted, which is still in the back of my mind as a very cool way to get multi-protocol support into Mailman. - Does our "NFS-safe" locks impose too much complexity and overhead to be worth it? Does anybody actually /use/ Mailman over NFS? Don't we sorta suspect the LockFile implementation anyway? Would we be better off using kernel file locks, or thread locks if we go to a MT model? Okay, now I'm rambling. What is the lowest hanging fruit that we might be able to attack? I'm up for any other ideas people have. -Barry [1] wget -E -H -k -p -nH -nd -Pdownload <url> followed by a little Python script to multipart/related it

3 4

a traceback
by Fil 14 May '03

14 May '03

When I "confirm" by email a spam received on a list (I want to discard it, of course, but Mailman writes "confirm", which is a biut weird, but thats' not my point today), I get this error in the log. And in fact the mail is not discarded. ==> /var/local/mailman/logs/error <== May 14 15:42:42 2003 (767) Uncaught runner exception: ASCII decoding error: ordinal not in range(128) May 14 15:42:42 2003 (767) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 105, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 155, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/CommandRunner.py", line 232, in _dispose res.process() File "/usr/local/mailman/Mailman/Queue/CommandRunner.py", line 105, in process stop = self.do_command(cmd, args) File "/usr/local/mailman/Mailman/Queue/CommandRunner.py", line 130, in do_command return self.do_command(cmd, args) File "/usr/local/mailman/Mailman/Queue/CommandRunner.py", line 132, in do_command return handler.process(self, args) File "/usr/local/mailman/Mailman/Commands/cmd_confirm.py", line 83, in process if line.lstrip() == match: UnicodeError: ASCII decoding error: ordinal not in range(128) May 14 15:42:42 2003 (767) SHUNTING: 1052919760.819023+d6d1f526383cb09df38958ab27e1c2cf00ef988b ==> /var/local/mailman/logs/vette <== May 14 15:42:42 2003 (767) spip-ann: Discarded posting: From: webdevelope1r(a)zipmail.com.br Subject: Faca de seu penis um PÊNIS -- Fil

1 0

Re: [Mailman-Developers] message data storage
by Donn Cave 13 May '03

13 May '03

Quoth Charlie Clark <charlie(a)begeistert.org>: ... | wow, that is a lot of traffic and a sluggish system. But I don't see how | 6500 mails can generate a 68 MB unless there are a lot of attachments. Even with attachments it takes some doing. The articles database actually tends to be double the size of the .txt archive for the same month. Not only this particular list. | As someone else has noted apart from the performance issue there is also | one of usability: searching 6500 messages is a proverbial needle in a | haystack. I don't know what the Next Generation archiver is but I guess | it's a move towards a more sophisticated persistance system which might | allow things like full-text searches. If I understand you correctly you | want to archive mails directly as they come in and keep the archiver / | db-connection open pretty much all the time. This would seem about the best | solution in the short term. Anything else sounds like: database to pass the | memory issue to something which is designed to handle it. Our own | experience with lists with a large member base is that Mailman isn't that | efficient at dealing with them which is why we're using an RDBMS adapter | and letting the RDBMS be the muscle while Mailman remains the brains. | | Novice question? How easy would it be just to dump to a database rather | than using Python storage? I don't know if anyone has a clear picture of this next generation system, it's just the subject of a dormant mailing list http://rogue.amk.ca/mailman/listinfo/ng-arch An external database might be a part of that picture, for all I know. I would expect that if a database really would pay off, it would take some rewriting anyway. It would be trivial to use a database for storage, but that would only slow things down. Really need to find a way to use the database more inside the threading algorithm. Donn Cave, donn(a)u.washington.edu

2 1

domain definition syntax
by marisa＠stat.umn.edu 12 May '03

12 May '03

I would like to permit everyone from my site (stat.umn.edu) to write to some mailing lists wihout moderator's approval. I tested several expressions on the "Addresses of members accepted for posting to this list without implicit approval requirement" but so far I only got individual addresses such as "marisa(a)stat.umn.edu" working. I may be using the wrong syntax. I did several tests using "*(a)stat\.umn\.edu" or "^.*(a)stat.umn.edu" and other combinations in that field but no one worked. Can you please help me with the syntax or tell me what else I may be doing wrong? Or how to get around with this, like, for example, if it makes sense to place everyone on my site in the "Addresses of members accepted for posting to this list without implicit approval requirement"? -- Seems drastic but it may work! We are running Mailman version 2.0.8. Thank you. -- Marisa Riviere U of M - Statistics marisa(a)stat.umn.edu 612-624-5859

2 1

mailmanctl
by Mike Burton 12 May '03

12 May '03

When I try to upgrade from 2.1.1 to 2.1.2, I get the following error when trying to start mailmanctl... Traceback (most recent call last) File "/usr/local/mailman/bin/qrunner", line 77, in ? from Mailman.i18n import _ File "/usr/local/mailman/Mailman/i18n.py", line 19, in ? import gettext File "/usr/local/lib/python2.2/gettext.py", line 49, in ? from errno import ENOENT Import Error: No module named errno I get several of these on startup. Python was installed from source code. Any ideas? Thanks, Mike

3 2