Mailman 3 November 1999 - Mailman-Developers

Changes working, and list moved
by bwarsaw＠python.org 12 Nov '99

12 Nov '99

Hey folks, the purpose of this message is twofold. First, it's to let you know that I feel pretty confident about the new changes, at least insofar as they system seems to be working pretty well. I don't yet know what impact, if any there will be on performance. Second, I've moved this mailing list to our new server so at least it should /seem/ faster (the new one is an Ultra2, the old one is a Sparc5). I'll be keeping an eye on this list over the next couple of days to make sure the new version is working properly. If so, then I'll probably start moving all the existing python.org mailing lists over to it (once I've verified our sysadmins are actually backing up the Ultra2). So you guys are the guinea pigs for the new changes. You should not notice any differences; the email addresses are unchanged, as are the URLs. You'll notice the new hostname leak in a couple of places (e.g. as URLs for the Mailman logo and the public archives), but those aren't important enough to fix at the moment. If you notice /anything/ strange going on, first try to email this list. If you suspect that the messages aren't getting through, email me directly at bwarsaw(a)python.org. Okay, have fun! -Barry

1 0

Changes working, and list moved
by bwarsaw＠python.org 12 Nov '99

12 Nov '99

Hey folks, the purpose of this message is twofold. First, it's to let you know that I feel pretty confident about the new changes, at least insofar as they system seems to be working pretty well. I don't yet know what impact, if any there will be on performance. Second, I've moved this mailing list to our new server so at least it should /seem/ faster (the new one is an Ultra2, the old one is a Sparc5). I'll be keeping an eye on this list over the next couple of days to make sure the new version is working properly. If so, then I'll probably start moving all the existing python.org mailing lists over to it (once I've verified our sysadmins are actually backing up the Ultra2). So you guys are the guinea pigs for the new changes. You should not notice any differences; the email addresses are unchanged, as are the URLs. You'll notice the new hostname leak in a couple of places (e.g. as URLs for the Mailman logo and the public archives), but those aren't important enough to fix at the moment. If you notice /anything/ strange going on, first try to email this list. If you suspect that the messages aren't getting through, email me directly at bwarsaw(a)python.org. Okay, have fun! -Barry

1 0

Massive changes (Long)
by Barry A. Warsaw 10 Nov '99

10 Nov '99

Folks, I have some rather massive changes to check into the CVS tree, which I'm going to explain below. Grab a cup of coffee and sit back. :) I should note that while I've done a lot of testing of this stuff, I'm sure I've missed things. I do not recommend using any of this on a production system, but I would definitely appreciate another set of eyes and hands taking a look. I think I have a way to selectively migrate lists to a new Mailman installation one at a time, so I will try some of my test lists on python.org. If all goes well, I may make y'all guinea pigs in the next round of experiments :) These changes were precipitated by the observation of occasional poor performance, and the intermittent reports of duplicate messages. I actually gathered enough data on python.org that implicate Mailman's built-in bulk mailer. So I wanted to experiment with alternative delivery strategies. In the process I got a new Ultra2 for python.org and decided it was time to play with a different MTA, so I sucked down Postfix and installed it instead of sendmail. What I wanted was an architecture that would let us drop in different delivery schemes easily, so that we could experiment with the best, and also so sites could easily add whatever delivery machinery made the most sense for them. I strongly suspect that no one scheme that Mailman coudl bundle will be right for everyone. For a while now, I've hated the tangle of code that gets called when a message is being delivered. So I've essentially ripped all that out for an architecture that I've talked about to a few people already. The idea is that when a message comes into the system, it is handled by a "pipeline". In Python terms, the pipeline is simply a list of modules, all of which have a function called `process' that takes two arguments: a MailList instance and a Message instance. MailLists you know. Messages are a subclass of rfc822.Messages with a little bit of extra stuff for convenience. Since rfc822.Message now supports a writeable interface, I was able to get rid of Message.IncomingMessage, Message.OutgoingMessage, and Message.NewsMessage. Here's the entire contents of MailList.Post(): # msg should be an Message.Message object. def Post(self, msg): self.IsListInitialized() # TBD: this is bogus and will later be configurable import Mailman.Handlers.HandlerAPI Mailman.Handlers.HandlerAPI.process(self, msg) self.Save() You'll see that that is a little bit shorter than what it used to be :). `Handlers' is a new package inside Mailman which contains all the modules for the pipeline. HandlerAPI.process() basically runs down the list of pipeline modules, calling each in turn. The specs for the <handler>.process() functions say that their return value is ignored, and they raise an exception if processing of the message is supposed to stop propagating. Here's the current list of pipeline modules for normal message delivery: pipeline = ['SpamDetect', 'Approve', 'Hold', 'Cleanse', 'CookHeaders', 'ToDigest', 'ToArchive', 'ToUsenet', 'CalcRecips', 'Decorate', 'Sendmail', 'Acknowledge', 'AfterDelivery', ] The idea here is that each pipeline module does one small thing and one thing only. For example, Approve.process() tries to figure out if the message has been pre-approved. It will also look for an Approved: header with a valid password. All communication between handlers happens via attributes on the message object, so in the case of the Approve handler, if it finds a valid "Approve: password" header, it sets "msg.approved = 1". Other modules (notably SpamDetect and Hold) will look for this attribute and short circuit if it finds a true value. In a similar vein, ToArchive and ToDigest will look for the `isdigest' attribute. If found and true, then the message is a digest so it won't be sent to the archive, nor will it be appended to the currently building digest. Now the neat thing is the Sendmail module. It's not clever at all, in fact it just popen's a sendmail (or sendmail-compatible, e.g. postfix) process, gives the recip addrs on the command line, and pipes the message to the proc's stdin. The one complication is that it splits up the recipients if necessary to keep the command line length manageable. What I think is neat is that you could implement an SMTPDeliver module which uses smtplib.py, or even convert to using Mailman's current bulk mailer (which I haven't done), or a delivery scheme of your own choosing. It should be quite easy to drop in just drop in this component while keeping everything else the same. The Sendmail module rqeuires that the msg object has an attribute called `recips' which is a list of addresses to send the message to. The CalcRecips module sets this attribute. A couple of things to notice. First, proper delivery of the message requires the correct order in the pipeline. Put Acknowledge before Sendmail and those who want acks of their messages would get them before the message was actually sent. Also, while this seems to work well for my simple tests, I think I need to play with more aspects under more realistic load to figure out whether this approach will work in the long run. Finally, there's the question of incoming messages (say injected from Usenet, or via the -request address). I think a pipeline will work well here too, e.g. I envision a pipeline of bounce detectors, but I'm not sure yet how this will integrate with the outgoing pipeline. A couple of other random but important notes. I've also significantly revamped the whole admin request stuff. Held messages and subscriptions are no longer kept in the list object. This should make your config.db files considerably smaller and faster to load, and will hopefully improve overall performance. All that information is now kept in a separate database file, which is currently implemented using Python's anydbm module (this might change). The good thing is that this file is only touched when a message or subscription is held, or when you hit the admindb web page. I've decided to finally make Python 1.5.2 the minimum supported platform. It's been out since April, is very stable, is available for all platforms (and I think there are RPMs available), and is much much MUCH better than any previous version of Python. I'm tired of trying to keep track of backwards compatibility issues. So the next version of Mailman will require at least Python 1.5.2. I should note that there will /not/ be a Python 1.5.3 -- Guido's decided that the next release will be 1.6, but that will definitely not be out (other than via the anonCVS tree) until next year some time. So... I'm going to check all these changes in so that I can continue testing. I'll hopefully also be ripping out lots more unused code. I'm not sure how much more time I'm going to be hacking on it, but I need to get this stuff working for python.org so at least until then. Apologies in advance to the I18N'ers for not getting to that stuff, and for making so many changes under the hood. If this stuff turns out to work well, I'll try to spend some time getting those mods integrated. Thanks, -Barry

1 1

Re: [Mailman-Developers] ABEND: problem with sync_members
by Richard Ellerbrock 09 Nov '99

09 Nov '99

>> Ok, I have found the problem. The check routines that validate an e-mail >address does not pick up addresses with spaces in them. Thus, the following >will cause an ABEND: >> >> richarde. test(a)eskom.co.za > >For what it's worth, that's not an RFC-compliant address. Any addresses >with spaces need to be enclosed within quotations, i.e > > "richarde. test"@eskom.co.za Correct. I have written a routine that grabs addresses from a database. It is rather strange what people enter as e-mail addresses. I then sync this to a mailing list. What I have done to get around the problem is to modify my SQL statement to exclude addresses with spaces in them. Also works :-) I had to modify sync_members to accept input from stdio though. -- Richard Ellerbrock richarde(a)eskom.co.za

1 0

Re: [Mailman-Developers] ABEND: problem with sync_members command
by Richard Ellerbrock 09 Nov '99

09 Nov '99

Ok, I have found the problem. The check routines that validate an e-mail address does not pick up addresses with spaces in them. Thus, the following will cause an ABEND: richarde. test(a)eskom.co.za Easy to reproduce. It must be in the code that checks e-mail addresses against a regular expression. Well, I doubt that it will be fixed as I submitted a patch against sync_members to allow getting input from stdio as stated in the help for the command. Nothing as yet. Is anything happening with mailman development? -- Richard Ellerbrock richarde(a)eskom.co.za >>> Sean Reifschneider <jafo(a)tummy.com> 1999/11/08 08:45:18 >>> On Mon, Nov 08, 1999 at 03:44:59PM +0200, Richard Ellerbrock wrote: >This is with Mailman 1.1 freshly downloaded. sync_members does not >complain about any invalid addresses. The number of addresses that I >want to sync are over 3000! Could this be a problem? I don't know, but last week I created a new list and freshly inserted over 3000 addresses into it with no problems. Sean -- "Engineering Tablets? Does that mean if I swallow one, I'll be an engineer?" -- Evelyn Mitchell Sean Reifschneider, Inimitably Superfluous <jafo(a)tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software. _______________________________________________ Mailman-Developers maillist - Mailman-Developers(a)python.org http://www.python.org/mailman/listinfo/mailman-developers

2 1

Questionnaire For My Project
by Luyin Zhao 09 Nov '99

09 Nov '99

Dear Mailman, I'm a graduate student (computer science) in University of Nebraska-Lincoln. The Questionnaire for open source testing below is for a paper I'm working on. Hope you can spend five minutes to finish it in your busy time. Any of your effort will be greatly appreciated! Luyin Questionnaire for Open Source Testing **************************************************** 1. What programming language is used in developing your product "Mailman 1.1":? **************************************************** 2. How many developers were involved in your project? **************************************************** 3. When do you begin to do testing? a. Requirement Analysis Phase b. Functional design Phase c. Internal Design Phase d. Implementation Phase e. After Implementation Note: Functional design is to describe the features of a system from technique-oriented perspective compared with the user-oriented perspective of requirement analysis. Internal design is to model the implementation of system, such as class diagrams. Answer: **************************************************** 4. The time you spent in testing compared with the total time of development? a. <20% b. 20%-40% c. 40%-60% d. 60-80% e. >80% Answer: **************************************************** 5. How do you prepare for your testing (using plan or schedule)? a. Written plan which was modified several times b. One plan draft without further modification c. I prepare it in my brain d. I have no plan Answer: **************************************************** 6. Do you inspect the following items? a. Requirement Documentation (if you have) b. Design Documentation (if you have) c. Source code (besides relying on compiler and program execution to find faults in code) Answer: **************************************************** 7. There are some defects resulted from the unclear definition of requirements. In the total defects of this category you found, approximately what percentage were found in code inspections, in unit tests and in system tests, respectively? a. <20% b. 20%-40% c. 40%-60% d. 60-80% e. >80% f. Not applicable Answer: , , **************************************************** 8. The percent of time you spent in document inspection (compared with total development time) is: a. <20% b. 20%-40% c. 40%-60% d. 60-80% e. >80% f. Not applicable Answer: **************************************************** 9. How many people did you ask to check your documents or source code? Do you think it is useful? a. No b. A little bit c. Useful but not necessary d. They found lots of defects that I haven抰 found e. Absolutely necessary Answer: **************************************************** 10. Do you use testing tools during your development? The name of it? Answer: **************************************************** 11. What is the basis of your testing? a. Test whether system implemented what user need b. Test whether system implemented technical features c. Find faults d. All the above. Answer: **************************************************** 12. What strategy did you adopt in choosing your test cases? Please list them according to the importance of finding defects of your system (descending)? a. Choose sample of valid inputs b. Choose those inputs most likely to cause failure c. Choose inputs according to your experience d. Other (specify ) Answer: **************************************************** 13. If you do coverage control, what percent of the code (lines of code) do you think are covered in your test? a. <20% b. 20%-40% c. 40%-60% d. 60-80% e. >80% f. Not Applicable Answer: **************************************************** 14. In defects you found, what percent of the following problem categories should be found in requirements phase? in design phase? in unit test ? in integration test? a. <20% b. 20%-40% c. 40%-60% d. 60-80% e. >80% F. Not Applicable Answer: **************************************************** 15. In which areas did you find most of the defects when you integrate and sell products (someone sell open source products such as RedHat)? a. User interface-related problems b. Database-related problems c. Network-related problems d. Logical e. Other(specify ) f. Not Applicable Answer: Luyin Zhao 535 N23 Street, #10 Lincoln, NE 68503 (402)438-9511

1 0

ABEND: problem with sync_members command
by Richard Ellerbrock 08 Nov '99

08 Nov '99

When I want to sync a huge list of e-mail addresses into a mailing list, I get the following ABEND: [root@lisa bin]# /home/mailman/bin/sync_members -w=no -a=no -f /tmp/tt test Traceback (innermost last): File "/home/mailman/bin/sync_members", line 257, in ? main() File "/home/mailman/bin/sync_members", line 239, in main mlist.ApprovedAddMember(addr, pw, digest, welcome, notifyadmin) File "/home/mailman/Mailman/MailList.py", line 944, in ApprovedAddMember raise e, v Mailman.Errors.MMBadEmailError [root@lisa bin]# This is with Mailman 1.1 freshly downloaded. sync_members does not complain about any invalid addresses. The number of addresses that I want to sync are over 3000! Could this be a problem? -- Richard Ellerbrock richarde(a)eskom.co.za

2 1

running on nice
by Peter Gervai 08 Nov '99

08 Nov '99

Hello, I got angry at mailman/python eating up my resources all the time, so I have put some os.nice(10) into deliver,post and mailcmd scripts. Maybe others would find this appealing and would worth a configurable variable somewhere. bye, grin

1 0

Announcing Mailman 1.1
by Barry A. Warsaw 06 Nov '99

06 Nov '99

I have just uploaded Mailman version 1.1 to www.list.org, and this will hopefully soon be mirrored to ftp.gnu.org. This release is precipitated by the removal of all GIFs from the distribution. There have also been some performance improvements, although lots more can still be done. ;) See the NEWS file excerpt below for details of changes. Sadly, the internationalization effort has not yet been integrated (mostly due to time constraints of the core maintainers). Mailman, the GNU mailing list manager, is software to help manage email discussion lists, much like Majordomo and Smartmail. Unlike most similar products, Mailman gives each mailing list a web page, and allows users to subscribe, unsubscribe, etc. over the web. Even the list manager can administer his or her list entirely from the web. Mailman also integrates most things people want to do with mailing lists, including archiving, mail-to-news gateways, and so on. Mailman is writtein primarily in Python, with a little bit of C for security. See http://www.list.org or http://www.gnu.org/software/mailman/mailman.html for details and downloading. -Barry -------------------- snip snip -------------------- - All GIFs removed. See http://www.gnu.org/philosophy/gif.html for the reason why. - Improvements to the Pipermail archiver which make things faster. Primary change is that the .txt files are not gzip'd on every posted message. Instead, use the new cron script `nightly_gzip' to gzip the .txt file in batches (this means that the .txt file will lag behind the on-line archives a little). - From the C drivers programs, Python is invoked with the -S option. This tells Python to avoid importing the site module, which can improve start up time of the Python process considerably. Note that the command line script invocation has not been changed. - New configuration variables PUBLIC_EXTERNAL_ARCHIVER and PRIVATE_EXTERNAL_ARCHIVER which can contain a shell command string for os.popen(). This can be used to invoke an external archiver instead of the bundled Pipermail archiver. See Defaults.py for details. - new script `bin/find_member' which can be used to search for a member by regular expression. - More child processes are reaped, which should eliminate most occurances of zombie processes. - A few small miscellaneous bug fixes (including PR#99, PR#107) and improvements to the file locking algorithms. -------------------- snip snip -------------------- <p><a href="http://www.list.org/">Mailman 1.1</a> - the GNU mailing list manager, implemented primarily in Python. (05-Nov-1999)

2 1

Patch against sync_members
by Richard Ellerbrock 01 Nov '99

01 Nov '99

Here is a little patch against the sync_members program to allow it to understand the following syntax: sync_members -w=no -a=no -f - listname The sync members command did not understand the - filename (read from stdio), even though the help for sync_members says that it does read from stdio. --- sync_members Mon Nov 1 15:51:51 1999 +++ sync_members.eskom Mon Nov 1 15:30:48 1999 @@ -166,6 +166,11 @@ filemembers = fp.readlines() fp.close() +# filemembers = sys.argv[i:] + # if there were no arguments on the command line, then it's probably + # a user error. XXX this is questionable +# if not filemembers: +# usage(1, 'No addresses found on the command line') else: try: fp = open(filename) -- Richard Ellerbrock richarde(a)eskom.co.za

1 0