[Mailman-Developers] Mailman 2.0 usage notes...

[Barry A. Warsaw]

>>>>> "cvr" == chuq von rospach <chuqui@plaidworks.com> writes:

    cvr> One thing that would help would be if the Sendmail module got
    cvr> productionalized so it could be used instead of SMTPDirect,
    cvr> because then you could use the -deliverymode=defer option,
    cvr> which you can't on SMTPDirect because it disables some
    cvr> spamchecking.

I'd love for this to happen, but someone else is going to have to do
the heavy lifting.  It's just not that important to me since most MTAs
can be configured to defer DNS resolution for localhost relays (as I
think Michael Yount suggested for sendmail).

    cvr> First, there's a problem with the way queues are
    cvr> processed. qrunners uses:

    cvr> 	for file in os.listdir(xxxx)

    cvr> to read the queue. The order is undefined, but in practice,
    cvr> it's basically blatting it out how it's stored in the
    cvr> inode.

In MM2.1, we essentialy do the same os.listdir() but we shuffle the
files randomly before looping over them.  Also, the resource controls
in MM2.0 are there because I was worried about things like cyclic
garbage bloating the qrunner process.  Requiring Python 2.0 is a
double win here because it's got cyclic garbage collection and I can
use random.shuffle().  So I can get rid of the process limits and just
run the loop infinitely.  This combined with the parallelizability of
the qrunner processes, and the splitting of the queue directories
should, I think, solve the problems you're seeing.
    
    cvr> This also led to finding a problem in the bounce processing
    cvr> area. The bouncer works pretty well, but it has one flaw for
    cvr> which I don't have an easy answer.

Several comments.  I intend to add a feature to MM (tho' probably not
until 3.0) that would let users register mutiple addresses under a
single account.  Thus I could teach Mailman that anything coming from
barry@digicool.com or barry@wooz.org is really from me and should not
be held.  This would allow the bounce matching code to match disparate
email addresses -- at the cost of cooperation from the user or list
admin.

OTOH, matching "bwarsaw@python.org" with "bwarsaw@mail.python.org"
regardless of which is in the bounce and which is in the database,
ought to be easy to manage (and I'm a little surprised the code
doesn't already do this).  See SMART_ADDRESS_MATCH in Defaults.py for
a variable that controls this in other contexts.

Mind submitting a bug report about this?
    
    cvr> If I'm subscribed as "chuq@plaidworks.com", but for some
    cvr> reason the bounce comes back as "chuq@mail.plaidworks.com"
    cvr> (or vice versa, or if I'm forwarding mail in some other
    cvr> name), the bouncer will catch the bounce and try to process
    cvr> it, not find me, and log it as a "user not
    cvr> subscribed". Unless the admin is somehow post-processing the
    cvr> bounce logs, though, that bounce is never REALLY handled, so
    cvr> it bounces indefinitely, and the admin never knows.

MM should be keeping much better stats about its operation, e.g. # of
messages per list received, delivered, etc.  Include in this a report
about bounces that hit non-members.

    cvr> This also over time encourages queue clogging and wastes
    cvr> bandwidth and CPU and all of that -- and worse, list admins
    cvr> and site admins probably think everything is fine because the
    cvr> bouncing system is working and these "not a member" bounces
    cvr> are never reported anywhere.

In MM2.1, you could tune the bounce qrunner to work less often, giving
priority to the incoming and outgoing queues.

    cvr> On the other, other hand, you probably don't want to just
    cvr> blat all these at admins, since they'll tune them out. But
    cvr> some kind of nightly report of some sort is the tradeoff I'd
    cvr> make, I think, so admins could see continual bounce problems
    cvr> that need to be manually investigated.

Good idea!
    
-Barry