[Mailman-Developers] a patch to scale Cgi/admin.py

[Joshua Ginsberg]

What it sounds like you really want in order to minimize database I/O is
to implement an in-memory caching system on top of the various methods
of the MemberAdaptor. So you'd have per MySQLMemberAdaptor object a
dictionary keyed the same as the database table with dictionaries for
the various fields per subscriber. If there is a KeyError when trying to
access the dictionary, hit the database. If the database returns no
rows, then you raise NotAMemberError or return whatever may be
appropriate.

True, this would only be effective per connection or per post, but it
seems to be the most efficient means of maximizing scalability. YMMV.

-jag

On Sat, 2005-10-29 at 16:49 +0200, Fil wrote:
> I'm knee-deep in Mailman/Gui/admin.py and it really doesn't scale. 
> 
> I use a test-list of 300k addresses, and it's a bit more than 5 minutes to
> get it to answer (if the connection holds that long, of course).
> 
> It's particularly true when using the MySQLMemberAdaptor, where many things
> are not taken from memory but are reprocessed with MySQL queries. 
> For instance, the part that checks if the member is regular/digest fetches
> fetches all the data for each subscriber (more plainly said, it's in N^2).
> 
> Another bottleneck is the list of chunks that is computed and displayed (and
> sent to the client) - his list is quite long to compute, and as a user it's
> not that useful in general.
> 
> Last but not least, the search facility calls the mysql-db for each member,
> in order to extract her name and regexp it; and that's very long. Is wasn't
> able to find how to speed this up, and just disabled it in my system (but
> not in the patch provided below)
> 
> So here are a few small changes, that make a radical improvement (down to 45
> seconds from 4 minutes):
> 
> --- /home/fil/src_mailman/mailman/Mailman/Cgi/admin.py  2005-02-12 21:22:55.000000000 +0100
> +++ Mailman/Cgi/admin.py        2005-10-29 16:43:56.116988176 +0200
> @@ -876,6 +876,7 @@ def membership_options(mlist, subcat, cg
>              doc.addError(_('Bad regular expression: ') + regexp)
>          else:
>              # BAW: There's got to be a more efficient way of doing this!
> +            # yes please... this doesn't scale at all
>              names = [mlist.getMemberName(s) or '' for s in all]
>              all = [a for n, a in zip(names, all)
>                     if cre.search(n) or cre.search(a)]
> @@ -978,6 +979,8 @@ def membership_options(mlist, subcat, cg
>                    MemberAdaptor.BYADMIN : _('A'),
>                    MemberAdaptor.BYBOUNCE: _('B'),
>                    }
> +    # memorize the regular-or-digest list
> +    regular_or_digest = mlist.getRegularMemberKeys()
>      # Now populate the rows
>      for addr in members:
>          link = Link(mlist.GetOptionsURL(addr, obscure=1),
> @@ -1021,8 +1024,8 @@ def membership_options(mlist, subcat, cg
>          # This code is less efficient than the original which did a has_key on
>          # the underlying dictionary attribute.  This version is slower and
>          # less memory efficient.  It points to a new MemberAdaptor interface
> -        # method.
> -        if addr in mlist.getRegularMemberKeys():
> +        # method. (Modified by Fil to "cache" the result - useful for MySQLMemberAdaptor)
> +        if addr in regular_or_digest:
>              cells.append(Center(CheckBox(addr + '_digest', 'off', 0).Format()))
>          else:
>              cells.append(Center(CheckBox(addr + '_digest', 'on', 1).Format()))
> @@ -1113,7 +1116,7 @@ def membership_options(mlist, subcat, cg
>          range listed below:</em>''')
>          chunkmembers = buckets[bucket]
>          last = len(chunkmembers)
> -        for i in range(numchunks):
> +        for i in range(min(10,numchunks)):
>              if i == chunkindex:
>                  continue
>              start = chunkmembers[i*chunksz]
> 
> 
> -- Fil
> 
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> http://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
> Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
> Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/jag%40fsf.org
> 
> Security Policy: http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp
> 
-- 
Joshua Ginsberg <jag at fsf.org>
Free Software Foundation - Senior Systems Administrator
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/mailman-developers/attachments/20051030/658dc4e8/attachment.pgp