Mailman 3 October 2005 - Mailman-Developers

XMLRPC Patch
by Joshua Ginsberg 13 Feb '06

13 Feb '06

I'm developing a patch to add an XMLRPC-based management interface to Mailman. Would this be something that you would be interested in trying to incorporate in the 2.1.x branch? Thanks! -jag -- Joshua Ginsberg <jag(a)fsf.org> Free Software Foundation - Senior Systems Administrator

5 14

MM3?
by Mike Burton 17 Nov '05

17 Nov '05

Hi folks, I have been watching with anticipation, but it's been a while since I have heard anything on MM3. Any noteworthy updates? Take care, Mike

3 3

an issue with MySQLMemberAdaptor
by Fil 31 Oct '05

31 Oct '05

Hi, I'm continuing my testing of the MySQLMemberAdaptor, and I found out that, if the database is down when someone wants to subscribe (or confirm subscription), for example, the messages are "shunted". This could be a problem when doing maintenance. It would be nice if Mailman could fwd the shunted messages to the listserv administrator, so she can decide what to do with it (i.e., reinstate the connection, and unshunt the files -- not just forget that the shunt directory exists). -- Fil

1 0

a patch to scale Cgi/admin.py
by Fil 31 Oct '05

31 Oct '05

I'm knee-deep in Mailman/Gui/admin.py and it really doesn't scale. I use a test-list of 300k addresses, and it's a bit more than 5 minutes to get it to answer (if the connection holds that long, of course). It's particularly true when using the MySQLMemberAdaptor, where many things are not taken from memory but are reprocessed with MySQL queries. For instance, the part that checks if the member is regular/digest fetches fetches all the data for each subscriber (more plainly said, it's in N^2). Another bottleneck is the list of chunks that is computed and displayed (and sent to the client) - his list is quite long to compute, and as a user it's not that useful in general. Last but not least, the search facility calls the mysql-db for each member, in order to extract her name and regexp it; and that's very long. Is wasn't able to find how to speed this up, and just disabled it in my system (but not in the patch provided below) So here are a few small changes, that make a radical improvement (down to 45 seconds from 4 minutes): --- /home/fil/src_mailman/mailman/Mailman/Cgi/admin.py 2005-02-12 21:22:55.000000000 +0100 +++ Mailman/Cgi/admin.py 2005-10-29 16:43:56.116988176 +0200 @@ -876,6 +876,7 @@ def membership_options(mlist, subcat, cg doc.addError(_('Bad regular expression: ') + regexp) else: # BAW: There's got to be a more efficient way of doing this! + # yes please... this doesn't scale at all names = [mlist.getMemberName(s) or '' for s in all] all = [a for n, a in zip(names, all) if cre.search(n) or cre.search(a)] @@ -978,6 +979,8 @@ def membership_options(mlist, subcat, cg MemberAdaptor.BYADMIN : _('A'), MemberAdaptor.BYBOUNCE: _('B'), } + # memorize the regular-or-digest list + regular_or_digest = mlist.getRegularMemberKeys() # Now populate the rows for addr in members: link = Link(mlist.GetOptionsURL(addr, obscure=1), @@ -1021,8 +1024,8 @@ def membership_options(mlist, subcat, cg # This code is less efficient than the original which did a has_key on # the underlying dictionary attribute. This version is slower and # less memory efficient. It points to a new MemberAdaptor interface - # method. - if addr in mlist.getRegularMemberKeys(): + # method. (Modified by Fil to "cache" the result - useful for MySQLMemberAdaptor) + if addr in regular_or_digest: cells.append(Center(CheckBox(addr + '_digest', 'off', 0).Format())) else: cells.append(Center(CheckBox(addr + '_digest', 'on', 1).Format())) @@ -1113,7 +1116,7 @@ def membership_options(mlist, subcat, cg range listed below:</em>''') chunkmembers = buckets[bucket] last = len(chunkmembers) - for i in range(numchunks): + for i in range(min(10,numchunks)): if i == chunkindex: continue start = chunkmembers[i*chunksz] -- Fil

2 4

Mysql MemberAdaptor 1.61 and Mailman 2.1.6
by Adrian Wells 28 Oct '05

28 Oct '05

Hello all. I have been working with Kev Green's MysqlMemberships.py Adaptor version 1.61 (SourceForge.net Mailman patch ID 839386) and Mailman 2.1.6 with Python 2.3.4. So far the adaptor has worked fairly well (with a minor patch to deal with one of the change made in Mailman 2.1.4, I believe, and the addition of a test to prevent the creation of unnecessary tables in the MySQL database - so this isn't a completely plain 1.61 version). However, there appears to be a bug/compatibility issue with this adapter concerning setting bounce information for list members. This process appears to be handled by a function called setBounceInfo in the adaptor. As it works now, setBounceInfo only successfully sets initial values in the MySQL database but fails to update subsequent bounce information. The bounce log seems to support these findings. It will report that the score was set to 2.0 but the MySQL database will still show a score of 1.0 for a bouncing list member... as a result members cannot receive bounce scores higher than 1.0! I am not proficient in Python and don't completely understand how Mailman operates so I'm interested in finding some help to understand how information generated by registerBounce in Bouncer.py is supposed to reach setBounceInfo in MysqlMemberships.py. Even a general understanding of how bounce information is processed in Mailman would be helpful for investigating this. Thank you, -Adrian

5 15

Issues with archiving directory and OS limitations
by AE Somerville 25 Oct '05

25 Oct '05

Hello, I have recently come across a problem that prevents the creation of any new lists for our site. Problem manifests as an inability of the create list process being able to make the archiving directories. The number appears to be when the directory count approaches 32,000 separate directories. How did it happen? We have close to 15,000+ lists but the archive directories houses two directories per list normally: /var/mailman/archives/private/<listname> /var/mailman/archives/private/<listname>.mbox So the number of directories are essentially doubled and then Linux has trouble with having any more. My temp solution: I have altered Site.py line 52 to add the list name again into the path for the archives. This halved the number of directories in the /var/mailman/archives/private/ level and pushed the extra directories into their own named sub directory. Now we can create new lists again (in our situation we have the list population updated daily and the lists themselves are added/deleted as required) def get_archpath(listname, domain=None, create=False, public=False): if public: subdir = mm_cfg.PUBLIC_ARCHIVE_FILE_DIR else: subdir = mm_cfg.PRIVATE_ARCHIVE_FILE_DIR path = os.path.join(subdir, listname, listname) if create: _makedir(path) return path Related problems (from the 'fix'): 1. The HTML links are not working for the archive site, but it would be nice to have them functioning. 2. Possible larger ramifications from the alteration of this function that I cannot see yet. Advice from the folks who are a lot more familiar with mailman would be great to point us at a more eloquent solution. ------------------------------------ Antony Somerville Network Programmer / Project Manager: QUT AD Upgrade Project Network Applications Queensland University of Technology, Brisbane Australia Phone +61 7 38644434 Fax +61 7 38642921

4 3

Manage many subscriptions with one password
by Larry Howe 23 Oct '05

23 Oct '05

Hello All, I am looking to add a feature to mailman and would appreciate some advice on how and where to get started. I run a MM server where users can subscribe to up to 8 different lists, and this number will grow. It would be nice for them not to have to deal with that many subs. I see discussion of a centralized user database, which is great. Correct me if I'm wrong, but that is an MM3 feature, and MM3 is a ways off? As an alternate, I was considering something like this: - when the user authenticates to his options page, there is a section there listing all possible mailing lists, and showing to which ones the user is currently subscribed. - user can simply check off the ones he wants and hit submit, and he will be subscribed / unsubscribed to the appropriate lists. - the button for List my Other Subscriptions is starting down this path. The differences would be: - list unsubscribed as well as subscribed lists - allow sub/unsub via this page - eventually, I could see adding a "group" field to the mailing lists so that the user would see only those lists in the current group. If I attempt this, is the project interested? If so, which branch? Is only MM3 being developed right now? I assume the centralized user database would obsolete what I'm proposing here. If this is a sane thing to try, any technical / implementation tips welcome. Larry Howe

1 0

Re: [Mailman-Developers] [Mailman-Users] including other fields in Mailman membership info
by Brad Knowles 13 Oct '05

13 Oct '05

At 7:21 PM -0700 2005-10-12, Jon Dugan wrote: > Is there a way to get mailman to have more than just the name and > email fields? While I recognize this would complicate things -- I'd > really like to leverage the great interface, password mailing, > confirmation -- etc. all in a broader membership database. Mailman does not have a membership database. There are unsupported third-party patches to provide a database "member adapter", but that's not the same thing. > Separate from the above, is there a way to make a call within "Full > Personalization" to extract a custom piece of data to place into an > email? > > For example, if I do have to store the comapny name for each member in > an external database, is there a way to get it into each email when > sending? See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq03.037.htp>. -- Brad Knowles, <brad(a)stop.mail-abuse.org> "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety." -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 SAGE member since 1995. See <http://www.sage.org/> for more info.

1 0

structural problem with MemberAdapter
by John Dennis 12 Oct '05

12 Oct '05

Good news, bad news: good news is I've implemented a LDAP based membership adapter that implements the complete MemberAdapter interface (the previous LDAP adapter was incomplete and limited in application). In stand alone testing my adapter seems to work quite nicely. Skip to the last paragraph if you don't need or want the details. Now the bad news, I'm having trouble integrating it because of either structural problems with adapters or a lack of understanding of how adapters are supposed to work. I'm looking for guidance or insight. Here are the issues I've run into: * Adapters are discovered by the presence of extend.py in the list's directory. How does extend.py get into the list directory? The only way I could figure was manual copying after list creation, this isn't a viable mechanism for a running site. Also, while I can see the value of per list adapters that seems like a niche case, a more common scenario would be every list resident at a site would use the same adapter module. To solve this problem I added code in the MailList constructor to look for a mm_cfg variable defining a global adapter module. But list creation/instantiation is a two step process, MailList object construction in __init__, followed by MailList.Create(). The adapter is not loaded in __init__ unless a name is provided, which is only done after Create() has been called. Most importantly an adapter cannot correctly initialize itself until the work of Create() is complete. This is very important, but more on this later. This sequence in TestBase.py leaves the adapter unloaded and uninitialized, it does not work. mlist = MailList.MailList() mlist.Create('_xtest', 'test(a)dom.ain', 'xxxxx') This idiom below which is found in most of the code does work because the list has already been created, but now we're back to the chicken and egg problem of when does extend.py get into the list directory. mlist = MailList.MailList(listname, lock=0) This is a good moment to go back to the issue of per list adapter initialization (or rather the lack of it). After the mlist is instantiated as above it might very well call an adapter method, for example mlist.addNewMember(). But the adapter has never been called as part of list creation. If the adapter must create entities in a backend database to service the list data it has never been given that opportunity. Thus the adapter is dependent on asking the question "does this list exist in my database, if not create it" every time the adapter object is instantiated, this is not only a big performance problem but it completely fails with the calling sequence in TestBase.py. If you've followed along so far pat yourself on the back :-) here's the payoff :-) Summary: -------- Unless I'm out to lunch with a horrible misunderstanding I propose 3 changes: 1) Provide a site wide variable that specifies an adapter for every list which is overridden by the presence of extend.py in the list directory. If the site wide adapter is used then MailList.__init__() loads it and calls an initialization function in the module (extend()??, __init__()??) 2) At the end of MailList.Create() it should test for the existence of an adapter and if so it should attempt to call the adapters Create() method. 3) Just as the adapter is not informed of list creation is is not informed of list destruction either, calls need to be added to invoke Remove() as well. I'm happy to provide patches for all this, I'm not interested in suggesting anyone else do more work, I'm mostly interested in knowing if I'm on the right track or if I have a misunderstanding. PostScript (or a few more questions): ------------------------------------- Now after having fully implemented the adapter interface I have to admit I don't really understand what its buying you over the existing OldMemberAdapter. My initial thought was to capitalize on existing user information at a site, but given the way mailman data is structured (a set of lists, each list may contain both local and unknown foreign users, and user properties are per list) then there seems to be little value in intermingling site user data and mailman list data. Also, it was not clear how an adapter might implement just a subset of the methods via inheritance, I suppose it would copy the function pointers from the mlist._memberadapter into its own methods before resetting mlist._memberadapter to itself. yes/no? -- John Dennis <jdennis(a)redhat.com>

2 2

The Mailman FAQ wizard (temporarily unavailable)
by Barry Warsaw 12 Oct '05

12 Oct '05

The Python.org administrators have moved the website to our spiffy new server, however the Mailman FAQ wizard hasn't yet been moved. It should be shortly. Please try not to make any changes to the FAQ wizard for the next day or so. Currently, until your DNS updates, you'll be hitting the wizard on the old machine -- at least until it gets disabled. When the database gets migrated, your changes may get lost. Then you may not have access to the new one until your DNS gets updated. I'll send out another message when the FAQ wizard has been successfully migrated. -Barry

1 1

Mailman-Developers October 2005