I'm developing a patch to add an XMLRPC-based management interface to
Mailman. Would this be something that you would be interested in trying
to incorporate in the 2.1.x branch? Thanks!
-jag
--
Joshua Ginsberg <jag(a)fsf.org>
Free Software Foundation - Senior Systems Administrator
Hi,
I'm continuing my testing of the MySQLMemberAdaptor, and I found out that,
if the database is down when someone wants to subscribe (or confirm
subscription), for example, the messages are "shunted". This could be a
problem when doing maintenance.
It would be nice if Mailman could fwd the shunted messages to the listserv
administrator, so she can decide what to do with it (i.e., reinstate the
connection, and unshunt the files -- not just forget that the shunt
directory exists).
-- Fil
I'm knee-deep in Mailman/Gui/admin.py and it really doesn't scale.
I use a test-list of 300k addresses, and it's a bit more than 5 minutes to
get it to answer (if the connection holds that long, of course).
It's particularly true when using the MySQLMemberAdaptor, where many things
are not taken from memory but are reprocessed with MySQL queries.
For instance, the part that checks if the member is regular/digest fetches
fetches all the data for each subscriber (more plainly said, it's in N^2).
Another bottleneck is the list of chunks that is computed and displayed (and
sent to the client) - his list is quite long to compute, and as a user it's
not that useful in general.
Last but not least, the search facility calls the mysql-db for each member,
in order to extract her name and regexp it; and that's very long. Is wasn't
able to find how to speed this up, and just disabled it in my system (but
not in the patch provided below)
So here are a few small changes, that make a radical improvement (down to 45
seconds from 4 minutes):
--- /home/fil/src_mailman/mailman/Mailman/Cgi/admin.py 2005-02-12 21:22:55.000000000 +0100
+++ Mailman/Cgi/admin.py 2005-10-29 16:43:56.116988176 +0200
@@ -876,6 +876,7 @@ def membership_options(mlist, subcat, cg
doc.addError(_('Bad regular expression: ') + regexp)
else:
# BAW: There's got to be a more efficient way of doing this!
+ # yes please... this doesn't scale at all
names = [mlist.getMemberName(s) or '' for s in all]
all = [a for n, a in zip(names, all)
if cre.search(n) or cre.search(a)]
@@ -978,6 +979,8 @@ def membership_options(mlist, subcat, cg
MemberAdaptor.BYADMIN : _('A'),
MemberAdaptor.BYBOUNCE: _('B'),
}
+ # memorize the regular-or-digest list
+ regular_or_digest = mlist.getRegularMemberKeys()
# Now populate the rows
for addr in members:
link = Link(mlist.GetOptionsURL(addr, obscure=1),
@@ -1021,8 +1024,8 @@ def membership_options(mlist, subcat, cg
# This code is less efficient than the original which did a has_key on
# the underlying dictionary attribute. This version is slower and
# less memory efficient. It points to a new MemberAdaptor interface
- # method.
- if addr in mlist.getRegularMemberKeys():
+ # method. (Modified by Fil to "cache" the result - useful for MySQLMemberAdaptor)
+ if addr in regular_or_digest:
cells.append(Center(CheckBox(addr + '_digest', 'off', 0).Format()))
else:
cells.append(Center(CheckBox(addr + '_digest', 'on', 1).Format()))
@@ -1113,7 +1116,7 @@ def membership_options(mlist, subcat, cg
range listed below:</em>''')
chunkmembers = buckets[bucket]
last = len(chunkmembers)
- for i in range(numchunks):
+ for i in range(min(10,numchunks)):
if i == chunkindex:
continue
start = chunkmembers[i*chunksz]
-- Fil
Hello all. I have been working with Kev Green's MysqlMemberships.py
Adaptor version 1.61 (SourceForge.net Mailman patch ID 839386) and Mailman
2.1.6 with Python 2.3.4. So far the adaptor has worked fairly well (with
a minor patch to deal with one of the change made in Mailman 2.1.4, I
believe, and the addition of a test to prevent the creation of unnecessary
tables in the MySQL database - so this isn't a completely plain 1.61
version).
However, there appears to be a bug/compatibility issue with this adapter
concerning setting bounce information for list members. This process
appears to be handled by a function called setBounceInfo in the adaptor.
As it works now, setBounceInfo only successfully sets initial values in
the MySQL database but fails to update subsequent bounce information. The
bounce log seems to support these findings. It will report that the score
was set to 2.0 but the MySQL database will still show a score of 1.0 for a
bouncing list member... as a result members cannot receive bounce scores
higher than 1.0!
I am not proficient in Python and don't completely understand how Mailman
operates so I'm interested in finding some help to understand how
information generated by registerBounce in Bouncer.py is supposed to reach
setBounceInfo in MysqlMemberships.py. Even a general understanding of how
bounce information is processed in Mailman would be helpful for
investigating this.
Thank you,
-Adrian
Hello,
I have recently come across a problem that prevents the creation of any new
lists for our site.
Problem manifests as an inability of the create list process being able to
make the archiving directories. The number appears to be when the directory
count approaches 32,000 separate directories.
How did it happen?
We have close to 15,000+ lists but the archive directories houses two
directories per list normally:
/var/mailman/archives/private/<listname>
/var/mailman/archives/private/<listname>.mbox
So the number of directories are essentially doubled and then Linux has
trouble with having any more.
My temp solution:
I have altered Site.py line 52 to add the list name again into the path for
the archives. This halved the number of directories in the
/var/mailman/archives/private/ level and pushed the extra directories into
their own named sub directory. Now we can create new lists again (in our
situation we have the list population updated daily and the lists themselves
are added/deleted as required)
def get_archpath(listname, domain=None, create=False, public=False):
if public:
subdir = mm_cfg.PUBLIC_ARCHIVE_FILE_DIR
else:
subdir = mm_cfg.PRIVATE_ARCHIVE_FILE_DIR
path = os.path.join(subdir, listname, listname)
if create:
_makedir(path)
return path
Related problems (from the 'fix'):
1. The HTML links are not working for the archive site, but it would be
nice to have them functioning.
2. Possible larger ramifications from the alteration of this function
that I cannot see yet.
Advice from the folks who are a lot more familiar with mailman would be
great to point us at a more eloquent solution.
------------------------------------
Antony Somerville
Network Programmer / Project Manager: QUT AD Upgrade Project
Network Applications
Queensland University of Technology, Brisbane Australia
Phone +61 7 38644434 Fax +61 7 38642921
Hello All,
I am looking to add a feature to mailman and would appreciate some advice on
how and where to get started. I run a MM server where users can subscribe to
up to 8 different lists, and this number will grow. It would be nice for them
not to have to deal with that many subs. I see discussion of a centralized
user database, which is great. Correct me if I'm wrong, but that is an MM3
feature, and MM3 is a ways off?
As an alternate, I was considering something like this:
- when the user authenticates to his options page, there is a section there
listing all possible mailing lists, and showing to which ones the user is
currently subscribed.
- user can simply check off the ones he wants and hit submit, and he will be
subscribed / unsubscribed to the appropriate lists.
- the button for List my Other Subscriptions is starting down this path. The
differences would be:
- list unsubscribed as well as subscribed lists
- allow sub/unsub via this page
- eventually, I could see adding a "group" field to the mailing lists so that
the user would see only those lists in the current group.
If I attempt this, is the project interested? If so, which branch? Is only MM3
being developed right now? I assume the centralized user database would
obsolete what I'm proposing here.
If this is a sane thing to try, any technical / implementation tips welcome.
Larry Howe
At 7:21 PM -0700 2005-10-12, Jon Dugan wrote:
> Is there a way to get mailman to have more than just the name and
> email fields? While I recognize this would complicate things -- I'd
> really like to leverage the great interface, password mailing,
> confirmation -- etc. all in a broader membership database.
Mailman does not have a membership database. There are
unsupported third-party patches to provide a database "member
adapter", but that's not the same thing.
> Separate from the above, is there a way to make a call within "Full
> Personalization" to extract a custom piece of data to place into an
> email?
>
> For example, if I do have to store the comapny name for each member in
> an external database, is there a way to get it into each email when
> sending?
See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq03.037.htp>.
--
Brad Knowles, <brad(a)stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.
Good news, bad news: good news is I've implemented a LDAP based
membership adapter that implements the complete MemberAdapter interface
(the previous LDAP adapter was incomplete and limited in application).
In stand alone testing my adapter seems to work quite nicely.
Skip to the last paragraph if you don't need or want the details.
Now the bad news, I'm having trouble integrating it because of either
structural problems with adapters or a lack of understanding of how
adapters are supposed to work. I'm looking for guidance or insight. Here
are the issues I've run into:
* Adapters are discovered by the presence of extend.py in the list's
directory. How does extend.py get into the list directory? The only way
I could figure was manual copying after list creation, this isn't a
viable mechanism for a running site.
Also, while I can see the value of per list adapters that seems like a
niche case, a more common scenario would be every list resident at a
site would use the same adapter module. To solve this problem I added
code in the MailList constructor to look for a mm_cfg variable defining
a global adapter module. But list creation/instantiation is a two step
process, MailList object construction in __init__, followed by
MailList.Create(). The adapter is not loaded in __init__ unless a name
is provided, which is only done after Create() has been called.
Most importantly an adapter cannot correctly initialize itself until the
work of Create() is complete. This is very important, but more on this
later.
This sequence in TestBase.py leaves the adapter unloaded and
uninitialized, it does not work.
mlist = MailList.MailList()
mlist.Create('_xtest', 'test(a)dom.ain', 'xxxxx')
This idiom below which is found in most of the code does work because
the list has already been created, but now we're back to the chicken and
egg problem of when does extend.py get into the list directory.
mlist = MailList.MailList(listname, lock=0)
This is a good moment to go back to the issue of per list adapter
initialization (or rather the lack of it). After the mlist is
instantiated as above it might very well call an adapter method, for
example mlist.addNewMember(). But the adapter has never been called as
part of list creation. If the adapter must create entities in a backend
database to service the list data it has never been given that
opportunity. Thus the adapter is dependent on asking the question "does
this list exist in my database, if not create it" every time the adapter
object is instantiated, this is not only a big performance problem but
it completely fails with the calling sequence in TestBase.py.
If you've followed along so far pat yourself on the back :-) here's the
payoff :-)
Summary:
--------
Unless I'm out to lunch with a horrible misunderstanding I propose 3
changes:
1) Provide a site wide variable that specifies an adapter for every list
which is overridden by the presence of extend.py in the list directory.
If the site wide adapter is used then MailList.__init__() loads it and
calls an initialization function in the module (extend()??,
__init__()??)
2) At the end of MailList.Create() it should test for the existence of
an adapter and if so it should attempt to call the adapters Create()
method.
3) Just as the adapter is not informed of list creation is is not
informed of list destruction either, calls need to be added to invoke
Remove() as well.
I'm happy to provide patches for all this, I'm not interested in
suggesting anyone else do more work, I'm mostly interested in knowing if
I'm on the right track or if I have a misunderstanding.
PostScript (or a few more questions):
-------------------------------------
Now after having fully implemented the adapter interface I have to admit
I don't really understand what its buying you over the existing
OldMemberAdapter. My initial thought was to capitalize on existing user
information at a site, but given the way mailman data is structured (a
set of lists, each list may contain both local and unknown foreign
users, and user properties are per list) then there seems to be little
value in intermingling site user data and mailman list data.
Also, it was not clear how an adapter might implement just a subset of
the methods via inheritance, I suppose it would copy the function
pointers from the mlist._memberadapter into its own methods before
resetting mlist._memberadapter to itself. yes/no?
--
John Dennis <jdennis(a)redhat.com>
The Python.org administrators have moved the website to our spiffy new
server, however the Mailman FAQ wizard hasn't yet been moved. It should
be shortly.
Please try not to make any changes to the FAQ wizard for the next day or
so. Currently, until your DNS updates, you'll be hitting the wizard on
the old machine -- at least until it gets disabled. When the database
gets migrated, your changes may get lost. Then you may not have access
to the new one until your DNS gets updated.
I'll send out another message when the FAQ wizard has been successfully
migrated.
-Barry