Mailman 3 October 1999 - Mailman-Developers

Speeding up Mailman
by Barry A. Warsaw 29 Oct '99

29 Oct '99

Here's a bit of low-hanging fruit that might speed up Mailman. I don't know exactly by how much, but it works by starting Python with the -S option. On my Sparc boxes I've seen 4-8x improvement in the startup times of the Python executable. I've tested the patch in that I'm sure it works, but I haven't put it in the python.org operation system yet. I plan on doing that next. Note that this only works for the CGI and mail wrappers, not the bin scripts, but it should still help out a lot. Let me know if you give this a shot, and whether you see any improvement or not. Thanks to Jeremy Hylton for the inspiration. -Barry -------------------- snip snip -------------------- Index: common.c =================================================================== RCS file: /projects/cvsroot/mailman/src/common.c,v retrieving revision 1.18 retrieving revision 1.19 diff -c -r1.18 -r1.19 *** common.c 1999/07/12 20:32:59 1.18 --- common.c 1999/10/29 15:06:43 1.19 *************** *** 196,212 **** newenv[j] = NULL; /* Now put together argv. This will contain first the absolute path ! * to the python executable, then the absolute path to the script, ! * then any additional args passed in argv above. */ ! newargv = (char**)malloc(sizeof(char*) * (argc + 2)); ! newargv[0] = python; ! newargv[1] = (char*)malloc(sizeof(char) * ( strlen(scriptdir) + strlen(script) + 1)); ! strcpy(newargv[1], scriptdir); ! strcat(newargv[1], script); /* now tack on all the rest of the arguments. we can skip argv's * first two arguments because, for cgi-wrapper there is only argv[0]. --- 196,215 ---- newenv[j] = NULL; /* Now put together argv. This will contain first the absolute path ! * to the Python executable, then the -S option (to speed executable ! * start times), then the absolute path to the script, then any ! * additional args passed in argv above. */ ! newargv = (char**)malloc(sizeof(char*) * (argc + 3)); ! j = 0; ! newargv[j++] = python; ! newargv[j++] = "-S"; ! newargv[j] = (char*)malloc(sizeof(char) * ( strlen(scriptdir) + strlen(script) + 1)); ! strcpy(newargv[j], scriptdir); ! strcat(newargv[j], script); /* now tack on all the rest of the arguments. we can skip argv's * first two arguments because, for cgi-wrapper there is only argv[0]. *************** *** 218,227 **** * * TBD: have to make sure this works with alias-wrapper. */ ! for (i = 2; i < argc; i++) ! newargv[i] = argv[i]; ! newargv[i] = NULL; /* return always means failure */ (void)execve(python, &newargv[0], &newenv[0]); --- 221,230 ---- * * TBD: have to make sure this works with alias-wrapper. */ ! for (i=2, j++; i < argc; i++) ! newargv[j++] = argv[i]; ! newargv[j] = NULL; /* return always means failure */ (void)execve(python, &newargv[0], &newenv[0]);

1 0

Mailman installation problems
by Thomas Esamie 27 Oct '99

27 Oct '99

Hi, I'm attempting to install mailman 1.0 on a Mac OS X server and managed to install Python (1.5.2) without too many problems and I ran the ./configure script. When I get to the the "make install" stage for mailman it starts chugging along merrily until we get to the following sequence >for f in answer_majordomo_mail mailcmd mailowner post driver; \ >do \ > /usr/bin/install -c -m 644 $f /Local/Applications/mailman/scripts; \ >done >cc -c -I. -DPREFIX="\"/Local/Applications/mailman\"" -DPYTHON="\"/usr/local/bin/python\"" -DHELPFUL -g -O2 -DHAVE_STRERROR=1 -DHAVE_SETREGID=1 -DHAVE_SYSLOG=1 -DSTDC_HEADERS=1 -DHAVE_SYSLOG_H=1 -DGETGROUPS_T=gid_t -DHAVE_VSNPRINTF=1 ./common.c >./common.c:26: `PREFIX' undeclared here (not in a function) >./common.c:26: `scripts' undeclared here (not in a function) >./common.c:26: parse error before `;' >make[1]: *** [common.o] Error 1 >make: *** [install] Error 2 Well I had a look at common.c and there are these lines which appear to cause the problem #define SCRIPTDIR PREFIX ## "/scripts/" /* trailing slash */ #define MODULEDIR PREFIX /* no trailing slash */ const char* scriptdir = SCRIPTDIR; [note this line is line 26] PREFIX does not appear to be an environment variable and it is not explicitly set in common.h I know this is kind of scratchy stuff but any clues on what to do from here? Greetings, Thomas PS. I'm no guru so assume complete stupidity on my part in any explanation :-)

1 0

unable to edit my archive page
by Jerry Adlersfluegel 22 Oct '99

22 Oct '99

I am trying to edit my Archives Index Page with the web interface, and my changes aren't showing up. The archive is private, and works just fine. I just finished configuring ht://dig to index the private archive and now I want to put the search doodads on the archive page. Changes I make on the web form show up in ~/lists/listname/archives.html but never make it to ~/archives/private/listname/index.html, even after someone sends mail to the list. The index.html file is overwritten everytime a message is posted, which blows away anything I tried to add there. Just to see if the web-based editor worked, I was able to successfully edit the General list information page. I would appreciate any help in modifying this file. (Mailman 1.0 on a RedHat 6.0 system.) Thanks! -- Jerry Adlersfluegel

1 0

Re: bug report id 146
by Adrian Joseph 22 Oct '99

22 Oct '99

The problem proved to be with a particular message which could not be approved,rejected or discarded. I looked at the message and the sender seemed to be a little odd. So, I made the change below to ListAdmin.py and it seemed to work okay, I don't know if this might cause other problems. Adrian 160c160 < 'sender' : msg.GetSender(), --- > 'sender' : strquote(msg.GetSender()),

1 0

CVS Access
by Adrian Eyre 18 Oct '99

18 Oct '99

The webpage mentions CVS access to sources, but not a CVSROOT. Does anyone have this string? -------------------------------------------- Adrian Eyre <mailto:a.eyre@optichrome.com> Optichrome Computer Solutions Ltd Maybury Road, Woking, Surrey, GU21 5HX, UK Tel: +44 1483 740 233 Fax: +44 1483 760 644 http://www.optichrome.com --------------------------------------------

1 0

Multiple Deliveries
by Will Leaver 14 Oct '99

14 Oct '99

Howdy to all! We are running four lists with Mailman 1.0, Python ver.1.5.1, Linux Red Hat, 5.?, and Sendmail, 8.9.3 We installed Mailman late in August. We have upgraded the smtplib.py to the current version. We are still sending duplicates of every post, and sometimes up to three or four copies. We have searched the archives, and think we understand the problem, but we are lacking clear direction to resolve it. Can anyone help? Will Leaver Joe Whittaker MASLOM, Inc.

1 0

[lm@bitmover.com: Re: FW: Staging System - Requirements Meeting Recap]
by Larry McVoy 14 Oct '99

14 Oct '99

hey there, two things: (a) is mailman better than majordomo? I assume it must be right, or you wouldn't be putting all this effort into it. I'm using majordomo now and I keep meaning to look for something that has better spam filters. Forgive the stupid questions, but should I switch? (b) JC has told me that you are considering maybe using BitKeeper for mailman development. I had a call from a large company today and they sent in their requirements and I spent a bunch of time answering them. It might be worth it for you to skim through the questions and answers, I suspect there is some overlap. I'd love it if it turned out that BK was useful to you (and I'd be equally bummed to find out it didn't work for you; if that's the case I want to know why: if there are technical reasons for you not using it, those will almost certainly be fixed, you guys have similar issues to everyone else so that means if it doesn't work for you, it doesn't work for very many other people). Finally, if you have questions, you can mail to me and/or dev(a)bitmover.com . The dev alias is all the people hacking on BK right now. Cheers, --lm -----Forwarded message----- Hi folks, here are BitMover's responses to your requirements. We think BitKeeper might be a good fit for you, let us know what you think. > Open Discussion - Requirements > a.. Direct FTP access We provide SSH (secure remote shell), RSH (BSD remote shell), and SMTP (email) as transports for moving stuff around. We could do a FTP transport but it seems questionable given that ssh is faster and more secure. If the issue is Windows, no problem, we support ssh on Windows. Use it all the time, in fact, between Windows and Unix. Oh, I forgot, we also work in local and networked file systems (NFS, SMB) and do resyncs between Unix and Windows using SMB. > b.. Graphical view of the file tree -or- dynamic tree structure We actually don't have this right now. But see the checkin response. We have it on Windows because we integrate with their GUI glop, but that's not much help. If we get far enough along and this is a go/no go thing for you, and you're buying 10,000 seats (:-) then we'll write one. Seriously, it's on the list. > c.. Check-in and check-out facilities Got 'em. Both command line and graphical. The command line stuff is like so bk new file - check a file into the system for the first time - aliases: bk delta -i, bk ci -i, bk admin -i bk edit file - check out a file locked for modification - aliases bk co -l, bk get -e bk ci file - check in modifications to a file - aliases: bk delta You can do all of the above on the entire tree, a sub tree, an explict list: bk -r new - checks in everything it finds bk -r edit - locks everything bk -r ci -yComment - checks in everything But the best tool is citool, which finds everything you've modified, lists in a graphical tool, and shows you the diffs as you're doing the check in. See http://www.bitkeeper.com/citool.html > d.. Open ended client support allowing developers to use any editing > tool - FTP to local machine with check-out - edit with whatever tool you > wish - FTP upload to server with check-in That's exactly the right answer. Use any editor you want. What you get with BK is the revision history files locally. If you've worked with RCS or SCCS, in many respects, BK is like that with all the added stuff you need to have multiple copies of the revision history. So every playpen is a repository, you can lock, edit, checkin, browse, etc., everything locally. No need for network connectivity except when you want to update your tree or the other tree. Oh, and you can update "sideways". One developer can slurp in the changes in a another developer's tree directly, without going through the "master" or "shared" tree. In CVS terms, I don't have to go back to the master repository, I can talk directly to you. The hierarchical nature of the system is strictly a convention thing, you can resync from anywhere to anywhere. > e.. Ability to cluster files into "Change Sets" for version control and > configuration management Chuckle. We have that in spades. We don't let you move data between playpens unless you've put it in a changeset (yeah, you could get the diffs and apply them with patch, we can't stop you, but if you want the system to do it, you have to "commit" whatever you have done to a changeset). The normal way you do this is when you pop into citool, there will be N+1 entries, where N are the files with mods, and the Nth+1 entry is the ChangeSet file. If you comment the ChangeSet file, you are creating a ChangeSet. The "diffs" shown for the ChangeSet file are the comments you have just typed in on all the other files like so src/foo.c Fix a bug in arg processing src/foo.h Include getopt.h with the idea being that the ChangeSet comments should be the idea or concept you just did, while the per file comments are implementation details. > f.. Allow user selectable update reports on one or more branches of the > development system I think this is what we call a LOD, but I need clarification. What exactly do you want? > g.. Ability for CGI to be accessible and scriptable from a remote shell - > should support Mac/PC/Linux/Unix/etc. Is this a source management system issue? I don't understand. And one huge bummer - we don't support the Mac; the system depends pretty heavily on various Unix tools (sed, sh, tr, diff, etc), which we provide for Windows but haven't worked up the ompf to do on the Mac. If the Mac is a requirement, we can support MacOS X no problem, that's Unix, but MacOS 8 is a real drag. Let me know, that might be a show stopper. There are ways we can work around this, using appletalk - you could do the source management crud on Linux and the editing on a Mac. Definitely a hack but I can relate to the requirement - the Mac has a bunch of useful tools for Web folks. Long term we will do a port. But that isn't likely to happen in the next few months. > h.. Change Logs You get these for free, it's the ChangeSet comment history. > i.. Concurrent Development of files - requires intelligent merging agent > for coordinating updates from multiple developers (CVS does this now) We do this better than anyone. Period. Everyone else screws up your history. Here's how: you have two developers A and B. They both do CVS update and have the same tree. They both modify file foo.c. A checks in first, so B "lost the race" and has to merge. What CVS will check in is not B's changes, it's B's changes plus whatever B had to do to merge. That is **two events** collapsed into one. Why is this a big deal? Undo. Suppose B's stuff was good and A's stuff wasn't. We can reconstruct B's stuff without the merge. CVS can't. It lost the information. Look at http://www.bitkeeper.com/sccstool.html - what you are seeing is the race and the merge. I'm "lm" and "awc" is my Windows guy. We were working parallel and the graph that you are seeing is the revision history after we merged. 1.89 is the merge delta, 1.86.1.1 and 1.86.1.2 are my deltas. If I want to reconstruct just my work, I do an undo and tell it which branch I want gone. And we do all this automagically. That tree of changes is a straight line, no branches tree as far as the user is concerned. The branches are created automatically when changes are merged in, you never have to do that. > j.. Access Control for each user of the development environment: Per user > basis and Per file basis We don't do this because this is an operating system issue. How you achieve the same effect is to restrict access using standard Unix file permissions on the master repository. > k.. Provide secure remote access to environment either through secure CRT, > SSL, domain restrictions, etc. SSH. > l.. Ability to differentiate gear, products and templates for exclusive > use (for one private label) and public use (all private labels) I don't understand this requirement. > Roles and Responsibilities > Engineering > a.. Priviledged access (on a user by user basis) to everything on the site > (cgi, kernels, html, graphics, etc.) This done through the OS. If you have an account and you can read and write the files, then you can read and write the files. > b.. Utilize a versioning source control system (SCS) for updates We provide something which is file format compatible with SCCS, AT&T's original revision control system from the 70's. A lot of people question this, there are claims that the SCCS file format is worse than RCS. That's not true, in fact, the opposite is true. RCS has one potential advantage that they don't even use: they store the most recent delta as a clear file and all the previous ones as diffs (backward diffs for going up the trunk and forward diffs for going down the branches). If RCS stored an offset and a size in the file, then they could seek to to the clear text and write the file out in one system call. They don't do that so they end up reading the whole file anyway (or most of it, it depends). Whatever. Someone could modify RCS to do what I said and it would still suck. Here's why: . No checksum. SCCS checksums the file and verifies the checksum every time you get the file. BK adds an additional per delta checksum. The point is that you put your IP into a system; that system should guard that IP very carefully. RCS makes a performance vs integrity tradeoff which will end up screwin you in the long run. . Annotation. SCCS can trivially give you a copy of the file with each line prefixed by any combination of the revision/user/date which added that line. In addition, BK can check out a copy of the file with every line in every revision in the file. Think about that. You know that somebody changed something and you know the string they changed but you don't know when. In BK you can find out by doing this bk sccscat -mu foo.c | grep string . ChangeSets. SCCS is a changeset engine, people just don't know it. What that means is that I can edit a file on one branch and say "I also want that delta over there which is on a different branch". SCCS will happily include it. You can creat new deltas and include/exclude any arbitrary list of deltas. It may not be obvious, but this is really cool and has far reaching ramifications. Not only can RCS not do this, if you tried to kludge it in, it would grow the file linearly for each include. In SCCS, including or excluding a delta costs you about 4 bytes per included delta. In other words, you can construct multiple different views of your data and it doesn't cost you. Ping me in the conference call about this if I did a poor job explaining it, it is profoundly important. It's what makes SCCS a ChangeSet engine and what makes RCS not a ChangeSet engine. > c.. Each engineer works in an independant workspace (currently known as > playpens) Yup. Each engineer can have as many playpens as s/he wants. Each are fully independent and get this: NO NO NO environment variables. You work in a playpen by saying cd ~/playpen/src bk vi foo.c If you want to work in a different playpen, you just say cd ~/different_playpen/src bk vi bar.c Environment variables suck. Just say no. Dare to break the environment variable habit :-) > d.. Depending on the engineer's privileges - able to push edits to staging > server at the branch or global levels If they have a login and write permissions on the staging area, they can. I really felt no need to reinvent Unix file permissions. Yeah, this screws the NT people but they can emulate the same thing by limiting access to the machine which has the staging area. > Web Development > a.. Priviledged access (on a user by user basis) to limited content on the > site (html, graphics, etc.) > b.. Utilize a versioning source control system (SCS) for updates > c.. Each developer works in an independant workspace (currently known as > playpens) > d.. Depending on the developers privileges - able to push edits to staging > server at the branch or global levels Same as above. > Quality Assurance > a.. Utilizes a selective build configuration management system (CMS) that > allows Q/A to selectivey test and build portions of the site We're weak here in some regards. We don't support partial repositories. In other words, when you do a resync, you get the whole tree. To deal with this, you will naturally split your project up into chunks. Each of these chunks is a repository. So far so good, but the one bummer is when you want to share data between two repositories. We currently don't have any way for data to be in two different "chunks" (we call 'em projects) at the same time. This needs more explanation so please ping me during the conference call. > b.. Ability to rollback change sets and individual files Chuckle. You bet. The easiest (but somewhat slow) way to roll back is like so: suppose you want to roll back to ChangeSet 1.123 (which was conviently tagged with alpha2). $ bk resync -r..alpha2 master alhpa2-test That will create a repository which is identical to the master repository when alpha2 went in (by the way, the tag is just for clarity, anywhere you use a tag you can use a revision). The other way is suppose you had a tree with ... alpha2 - 1.124 - 1.125 and you wanted alpha2. You can dothis $ bk undo 1.124,1.125 and that will DESTROY those two changesets. You'd better have a copy of them somewhere if you want 'em back). > c.. Tie in builds with bug tracking system to complete a closed loop > tracking process Busted. We want to do this but this is a 2.0 or perhaps 3.0 before we have it. We have a plan for doing it but right now we don't have diddly. > d.. Test and submit builds for Go Live! ??? > Release Management > a.. Schedules and approves builds for go live > b.. Coordinates builds with Q/A for testing Seems OK. > Private Label Partners > a.. Restricted access to their branch > b.. Access control based on a case by case basis I don't know what these are. --------------------------------------------- One thing you forgot to ask about is file renames. This is another place where people fall down. We don't (surprise!). We handle file pathname changes identically to file content changes. Pathnames are revisioned and propogate. Consider a couple of test cases: You Me Resync you to me mv foo bar nothing moves foo to bar in my tree change foo mv foo bar applies change in foo to bar in my tree mv foo bar mv foo blech prompts you with name conflict This is not a big deal until you need it to work but then it is a huge deal. It can bring your develpment to a halt for days while your engineers unscramble the mess. We just make that problem go away. You also forgot file permissions. We pick up the permissions as of the time that the file was originally checked in "bk new". After that, if you want to change them, you just say "bk chmod 755 foo.sh" and it saves the modes. On windows, only the top bits (owner permissions) are used, but it doesn't stomp on the lower bits. > Possible Vendors > a.. Continuus - Continuus http://www.continuus.com/ > b.. Interwoven - Teamsite http://www.interwoven.com/ > c.. Clear Case - Need Vendor Name Rational Software, http://www.rational.com > d.. Bit Keeper - Need Vendor Name BitMover, Inc. And it is "BitKeeper" if you want to be picky. http://www.bitkeeper.com Also consider the following: CVS (it's free and I can show you what you could do to sort of have some of these features, not all, but some). Perforce, http://www.perforce.com - I know Chris, great guy, nice little tool. It doesn't do what you want but it is quite popular and you should know it exists. TrueChange from TrueSoft, http://www.truesoft.com - this is the only commercially available ChangeSet engine other than BitKeeper. Cheers, Larry McVoy President, BitMover, Inc. -----End of forwarded message----- -- --- Larry McVoy lm(a)bitmover.com http://www.bitmover.com/lm

3 2

Re: [Mailman-Developers] Plea for Help
by J C Lawrence 14 Oct '99

14 Oct '99

On Wed, 13 Oct 1999 23:26:53 -0400 (EDT) Barry A Warsaw <bwarsaw(a)cnri.reston.va.us> wrote: <<Regarding Aegis>> >>>>>> "JCL" == J C Lawrence <claw(a)varesearch.com> writes: JCL> There's been some brief discussion on Linux-Kernel (searchable JCL> archives at http://www.mail-archive.com/), but not a whole lot. JCL> I've never seen a real installation or user report from other JCL> than the authors. > And yet it appears to have been out since 1991. Interesting. That's, umm, one way to put it. -- J C Lawrence Home: claw(a)kanga.nu ---------(*) Work: claw(a)varesearch.com ... Beware of cromagnons wearing chewing gum and palm pilots ...

1 0

Re: [Mailman-Developers] Plea for Help
by J C Lawrence 14 Oct '99

14 Oct '99

On Wed, 13 Oct 1999 18:45:50 -0400 (EDT) Barry A Warsaw <bwarsaw(a)cnri.reston.va.us> wrote: > BTW, anybody have real-world experience with Aegis? It's GPL'd > but I've never heard of it before... There's been some brief discussion on Linux-Kernel (searchable archives at http://www.mail-archive.com/), but not a whole lot. I've never seen a real installation or user report from other than the authors. -- J C Lawrence Home: claw(a)kanga.nu ---------(*) Work: claw(a)varesearch.com ... Beware of cromagnons wearing chewing gum and palm pilots ...

2 1

Re: [Mailman-Developers] Plea for Help
by J C Lawrence 14 Oct '99

14 Oct '99

On Wed, 13 Oct 1999 18:40:53 -0400 (EDT) Barry A Warsaw <bwarsaw(a)cnri.reston.va.us> wrote: > Thanks for the info JC. The BK web pages seem pretty out of date > though, and I didn't see anything about downloads. Interestingly > enough I also didn't see any links to the supposedly public free > software change logs :) BK is not released *yet*. Access is currently restricted to people who sign the beta agreement etc. BK should be released fairly soon tho (Larry: What's the current predict?) > Everything else sounds interesting. I'd like to come up with a > solution that would eventually translate to all the projects we've > got here (JPython and Python) so I want to investigate Aegis, and > get Guido's feedback on things as well. Please note that I'm explicitly biased here. I've used and don't like CVS. I'm aware of Aegis only intellectually. Aegis follows a somewhat similar design model to BK, but with a much more overweaning and dictatorial (you will do things our way or you will not do anything so NYAH!) implementation. Compared to the various SCM tools I've used at HP, IBM, SGI, etc I'm damn near in love with BK, so no brownie points for guessing which I'd recommend. > I appreciate all the feedback, information and experience you guys > share. Absolutely. Some of the things I particularly like about BK: -- Cross platform. It is identical under Linux, Solaris, IRIX, Windows NT, etc. -- Resolves 95% of the three way merge problem. -- The GUI tools for merges (when they do happen) does almost exactly what you want a three-way merge tool to do. Some comments I wrote offlist comparing CVS and BK: -- BK removes 95% of the merge problem. That's enough alone to sell it to me. BK's merge tools are not perfect, but they're a sight better than anything else I've seen. -- BK gives you real audit trails for changes. CVS doesn't. No more CVSBLAME. -- BK handles file deletes and renames. CVS doesn't. -- BK allows you to stage work and to distribute only known-working source sets. CVS doesn't. -- Nothing in BK runs as root. CVS' pserver must run as root to allow UID changes, and pserver was not built with security in mind. -- BK can be run over SSH. CVS can run pserver connections over an SSH port forward, but its a pain (I've done it). CVS over SSH performance also sucks. BK over SSH is quite nippy. -- BK can use email, even PGP'ed email, as it transport. CVS can't. -- BK handles binary files transparently. CVS doesn't even pretend to. While BK's handling is not great (it checks in the UUencoded copy), it does work. -- Nothing happens to a BK repository that you don't explicitly Okay, and EVERYTHING can be rolled back to its prior state. Got a bum changeset in? No problem: Just back it out. There are few controls on what happens to a CVS repository other than user/passwd authentication, and absolutely no rollback capabilities. -- One of Larry's big items: BK is very very bandwidth light. You can easily (and often quickly) sync two massive repositories over a slow link. Basically under BK only the absolutely needed changes are sent, nothing else. CVS is not so generous. -- BK imposes no user heirachy. All BK work areas are first class repositories. Resync your work area from a repository and the result is bit identical to the other repository and anybody else can resync from you. CVS imposes a single central server/many-clients model. BK allows any sort of tree, or even mesh, to be used. -- BK handles branches fairly well, sorta. I'm still waiting for Larry's promised LinesOfDevelopment code so I can get a handle on what he's trying to do there. If it is what I understand it will be clean and fairly neat. CVS OTOH is just painful once you have anything more than a single branch. -- BK makes it very easy to seperate your work on a source tree into waht I call "projects" (eg a fix for bug X, new feature Y, etc), each distinct from the others and then lets you track and propagate them individually. While similar is possible under CVS, it is nowhere near as easy and the individual propagation just ain't there. -- J C Lawrence Home: claw(a)kanga.nu ---------(*) Work: claw(a)varesearch.com ... Beware of cromagnons wearing chewing gum and palm pilots ...

2 1

Mailman-Developers October 1999