Scott and developers,

I've got a couple of things to try re the duplicate delivery problem. I suspect it's related to the new queuing mechanism, and also to the linux permissions problem that i unravelled for corbett last week.

I suspect it's related because of the sensitivity to forking - and the fact that you're on a linux system. When forking on recent rh linux, you lose the effective UID needed to access the files already written in the queue directory, so the queued files remain around despite being successfully sent. The residual queue files then get processed next time around, yielding the duplicate messages. I don't have the time to verify this, but it sounds like a likely prospect to me. If it is in fact the cause, you might profit from the same workaround as for corbett's posting failure problems - set the ownership of the ~mailman/data directory to 'mail', or whatever ID your MTA runs with, and see if that helps.

Alternately, before i went away last week i refined the queuing mechanism a bit to take care of some particular exceptions that can foul the delivery. You might try substituting the following replacements for TrySMTPDelivery() and DeliverToUser() in ~mailman/Mailman/Utils.py, and using the attached version of ~mailman/scripts/contact_transport instead of the installed one, to get some error detection for uncooperative queue files.

NOTE that i don't have a vanilla 1.0b5 installation to test these against, so try them a bit at a time - i really can't guarantee that they're compatable, though the interface changes ought to be small enough to be ok...

If any of you try these things, please let me know what you find.

Ken

def DeliverToUser(msg, recipient, add_headers=[]): """Use smtplib to deliver message.

Optional argument add_headers should be a list of headers to be added
to the message, e.g. for Errors-To and X-No-Archive."""
# We fork to ensure no deadlock.  Otherwise, even if sendmail is
# invoked in forking mode, if it eg detects a bad address before
# forking, then it will try deliver to the errorsto addr *in the
# foreground*.  If the errorsto happens to be the list owner for a list
# that is doing the send - and holding a lock - then the delivery will
# hang pending release of the lock - deadlock.

if os.fork():
    return
sender = msg.GetSender()

try:
    try:
        msg.headers.remove('\n')
    except ValueError:
        pass
    if not msg.getheader('to'):
        msg.headers.append('To: %s\n' % recipient)
    for i in add_headers:
        if i and i[-1] != '\n':
          i = i + '\n'
        msg.headers.append(i)

    text = string.join(msg.headers, '')+ '\n'+ QuotePeriods(msg.body)
    import OutgoingQueue
    queue_id = OutgoingQueue.enqueueMessage(sender, recipient, text)
    TrySMTPDelivery(recipient,sender,text,queue_id)
    # Just in case there's still something waiting to be sent...
    OutgoingQueue.processQueue()
finally:
    os._exit(0)

def TrySMTPDelivery(recipient, sender, text, queue_entry): import sys, socket import smtplib import OutgoingQueue

try:
    con = smtplib.SmtpConnection(mm_cfg.SMTPHOST)
    con.helo(mm_cfg.DEFAULT_HOST_NAME)
    con.send(to=recipient,frm=sender,text=text)
    con.quit()
    dequeue = 1
    failure = None

# Any exceptions that warrant leaving the message on the queue should
# be identified by their exception, below, with setting 'dequeue' to 1
# and 'failure' to something suitable.  Without a particular exception
# we fall through to the blanket 'except:', which dequeues the message.

except socket.error:
    # MTA not responding, or other socket prob - leave on queue.
    dequeue = 0
    failure = sys.exc_info()

except:
    # Unanticipated cause of delivery failure - *don't* leave message 
    # queued, or it may stay, with reattempted delivery, forever...
    dequeue = 1
    failure = sys.exc_info()

if dequeue:
    OutgoingQueue.dequeueMessage(queue_entry)
if failure:
    # XXX Here may be the place to get the failure info back to the
    #     list object, so it can disable the recipient, etc.  But how?
    from Logging.StampedLogger import StampedLogger
    l = StampedLogger("smtp-failures", "TrySMTPDelivery", immediate=1)
    l.write("To %s:\n" % recipient)
    l.write("\t %s / %s\n" % (failure[0], failure[1]))
    l.flush()

#! /usr/bin/env python # # Copyright (C) 1998 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

"""Send a message via local SMTP, or queue it if SMTP port is not responding.

The script takes the following protocol on stdin:

line [1]: sender line [2:n+1]: n recipients line [n+2]: <empty> - delimiting end of recipients line [n+3:]: message content """

import sys, os import paths

# XXX: this really should be merged with Python's standard smtplib library from Mailman import mm_cfg from Mailman import smtplib from Mailman import Utils from Mailman import OutgoingQueue

from Mailman.Logging.Utils import LogStdErr LogStdErr("error", "contact_transport")

from_addr = sys.stdin.readline()[:-1] to_addrs = [] while 1: l = sys.stdin.readline()[:-1] if not l: break to_addrs.append(l) text = sys.stdin.read()

queue_id = OutgoingQueue.enqueueMessage(from_addr, to_addrs, text) Utils.TrySMTPDelivery(to_addrs, from_addr, text, queue_id) OutgoingQueue.processQueue()