[Mailman-Developers] Re: External subscriber lists in 2.1

[Barry A. Warsaw]

On Sun, 23 Dec 2001, Barry A. Warsaw wrote:
> Note that if it was too expensive for getRegularMemberKeys() to return
> an in-memory list, it could (if you use Python 2.2) return an iterator
> object that implemented things in a more efficient manner, e.g. by
> paging through blocks.  I believe that any place where we expect a
> Python sequence (list) we could probably accept an iterator.

    DN> Do you know if this is in fact the case?  Is this how I should
    DN> implement it?

Nope, never actually tried it.  Note that in Python 2.2 you might
actually want to try a generator, but again, I've not tried it at all.

    DN> (While I've hacked in python fairly regularly, I've not done
    DN> serious Python development since ~1996, so some of these
    DN> questions will be fairly basic "How's it work in Python2.2?" 
    DN> ones.  Of course, this brings up the question of whether or
    DN> not it's reasonable to require that--don't we currently only
    DN> require 2.1.3?)

Yes.  Mailman 2.1 will work with Python 2.1.3 and beyond.  The next
version will require at least Python 2.2.1.  So if you want to also
support Py2.1, iterators and generators are out.

    DN> Below are different ways the results of
    DN> "get{Regular,Digest}MemberKeys" and "getMembers" are used in
    DN> various places in the codebase.  Will each of them still work
    DN> if those methods return an iterator instead of a list?  (For
    DN> example, does len(iterator) work?)

len(iterator) doesn't work (unfortunately, IMO).  So maybe we're
screwed because we'd have to call list() on the thing to be able to
give it to len().  This actually is the cause of a few recent bugs
with email.Iterators.body_line_iterator().  OTOH, if the iterator
object we return has an __len__() method, we should be okay.

    DN> I'm guessing that a good answer might be that we want to add
    DN> more general-purpose accessor methods in class MemberAdaptor
    DN> (and have the rest of the codebase use those where
    DN> appropriate) so that if there's a more efficient way for a
    DN> specific backend to provide specific data, it is able to do
    DN> so.  I'll include the two methods I propose at the end of this
    DN> message.

You're probably right.

in Handlers/CalcRecips.py:
    # Calculate the regular recipients of the message
    recips = [mlist.getMemberCPAddress(m)
              for m in mlist.getRegularMemberKeys()
              if mlist.getDeliveryStatus(m) == ENABLED]
and:
            recips = mlist.getMemberCPAddresses(mlist.getRegularMemberKeys() +
                                                mlist.getDigestMemberKeys())

in Cgi/admin.py:
        if not mlist.nondigestable and mlist.getRegularMemberKeys():
and:
        if addr in mlist.getRegularMemberKeys():
and:
    # If there are more members than allowed by chunksize, then we split the
    # membership up alphabetically.  Otherwise just display them all.
    chunksz = mlist.admin_member_chunksize
    all = mlist.getMembers()
    all.sort(lambda x, y: cmp(x.lower(), y.lower()))
then:
            # BAW: There's got to be a more efficient way of doing this!
            names = [mlist.getMemberName(s) or '' for s in all]
            all = [a for n, a in zip(names, all)
                   if cre.search(n) or cre.search(a)]

in HTMLFormatter.py:
            members = self.getRegularMemberKeys()
            for m in members:
                if not self.getMemberOption(m, conceal_sub):
                    people.append(m)
            num_concealed = len(members) - len(people)
and:
        member_len = len(self.getRegularMemberKeys())


    | def getNumMembers(self, type, status, options, regexp=None):
    |     """Get the number of members of this mailing list matching type and
    |     status, and optionally matching the regular expression passed in.

Would it be good enough to do something like the following:

def getMatchingMembers(self, func):
    """Return a list of members for which function evaluates true.

    For each member in the database, call func(), passing in the
    member's subscribed address.  If func() returns true, the address
    is included in the returned list.
    """

likewise,

def getMatchingMembersCount(self, func):
    """Return a count of the members for which function evaluates true.

    For each member in the database, call func(), passing in the
    member's subscribed address.  If func() returns true, that member
    is included in the count.
    """

This might be a little less efficient than your APIs because of the
function call, but it's more flexible.  OTOH, it might be too flexible
(it'd be hard to run this in a database or translate to selects,
etc.), so I'm willing to be persuaded.

Also getMatchingMembers() would have to return a list for Py2.1
compatibility, but could return an iterator or generator in Py2.2+.

-Barry