I just have a question.... Peering through the queueing system, Inote that there does not seem to be any locking of the queuefiles... Correct? The way it appears to me that mailman is working is thus: every outgoing message is queued, then an attempt to feed it to the MTA is made. After that it tries to run the queue to catch anything waiting there. What keeps another process running the queue from grabbing & delivering the queued message whilst the first proccess is trying to send the same message to the MTA? Especially whence you have several proccesses rapidly forked all doing delivery at once?
Here's the situation I'm worried about:Proccess 1 queues the message, and attempts to feed it to the MTA Proccess 2 running queue reads the queued message, and also attempts to deliver it to the MTA. Proccess 1 succeeds in in delivering the message and deletes it from the queue. Proccess 2 ALSO succeeds in delivering the message, creating a duplicate. Proccess 2 tries to delete the message from the queue, and can't since proccess 1 already deleted it, generating this traceback, which I _have_ seen in my error log:
Aug 29 04:10:59 1998 contact_transport: Traceback (innermost last): contact_transport: File "/usr/services/mailman/scripts/contact_transport", line 52, in ? contact_transport: OutgoingQueue.processQueue() contact_transport: File "/usr/services/mailman/Mailman/OutgoingQueue.py", line 38, in processQueue contact_transport: Utils.TrySMTPDelivery(recip,sender,text,full_fname) contact_transport: File "/usr/services/mailman/Mailman/Utils.py", line 230, in TrySMTPDelivery contact_transport: OutgoingQueue.dequeueMessage(queue_entry) contact_transport: File "/usr/services/mailman/Mailman/OutgoingQueue.py", line 25, in dequeueMessage contact_transport: os.unlink(msg) contact_transport: os . error : (2, 'No such file or directory')
Having pointed out a possible problem, perhaps I can suggest a possible solution? Howabout this:
Whenever Mailman goes to deliver mail, it dosen't actually deliverit. Rather, it just queues the message. Instead, there is a single, separate proccess that is kept running, and all it does is dequeue messsages. It would keep a PID file and touch it periodically, so you could run a cron job to make sure it is still running. Since it would hold no locks on any lists, it wouldn't have to worry about forking to avoid deadlocks, and since it would be run under the mailman uid it wouldn't be affected by any possible setgid weirdness (i.e. w/ linux)).
If anyone is interested in this, let me know, & I'll putsomething together. It should be really quick.
-The Dragon De Monsyne
On Sat, 29 Aug 1998, The Dragon De Monsyne wrote:
I just have a question.... Peering through the queueing system, I note that there does not seem to be any locking of the queuefiles... Correct? The way it appears to me that mailman is working is thus: every outgoing message is queued, then an attempt to feed it to the MTA is made. After that it tries to run the queue to catch anything waiting there. What keeps another process running the queue from grabbing & delivering the queued message whilst the first proccess is trying to send the same message to the MTA? Especially whence you have several proccesses rapidly forked all doing delivery at once?
John will have the authoritative answer on the queueing mechanism (though i've messed with, and contributed some refinements to it), but we have seen exactly the behavior you describe, where messages are delivered out from under a TrySMTPDelivery process. The solution you describe - a separate process, _plus_ a cron job to ensure it's always going - sounds a lot more cumbersome than simply using file locks, as you mention.
(It occurred to me to wonder whether it's best to just have the absence of the queue file ignored, with the assumption that some other process did the delivery. But then i realized that this may in some cases mean duplicate deliveries, which again calls for file locking.)
Note also that in the context of a persistent server, your idea of a process to handle the queue - probably a thread in the server - will make a lot of sense. But until that time, more processes hanging around strike me as undesirability administrative and conceptual complexity...
ken manheimer klm@python.org
On Sat, 29 Aug 1998, Ken Manheimer wrote:
On Sat, 29 Aug 1998, The Dragon De Monsyne wrote:
I just have a question.... Peering through the queueing system, I note that there does not seem to be any locking of the queuefiles... Correct? The way it appears to me that mailman is working is thus: every outgoing message is queued, then an attempt to feed it to the MTA is made. After that it tries to run the queue to catch anything waiting there. What keeps another process running the queue from grabbing & delivering the queued message whilst the first proccess is trying to send the same message to the MTA? Especially whence you have several proccesses rapidly forked all doing delivery at once?
John will have the authoritative answer on the queueing mechanism (though i've messed with, and contributed some refinements to it), but we have seen exactly the behavior you describe, where messages are delivered out from under a TrySMTPDelivery process. The solution you describe - a separate process, _plus_ a cron job to ensure it's always going - sounds a lot more cumbersome than simply using file locks, as you mention.
I don't see why it would be cumbersome at all... All ye'd havewould be 1 script you would run. It would check a pid file to make sure it's not already running, so you could run it from a cronjob to have it automatically restarted if need be. Sounds alot less cumbersome than all the elaborate queueing, forking, queue-running, and now file-locking mailman otherwise need to do.
-The Dragon De MonsyneI don't see why it would be cumbersome at all... All ye'd have would be 1 script you would run. It would check a pid file to make sure it's not already running, so you could run it from a cronjob to have it automatically restarted if need be. Sounds alot less cumbersome than all the elaborate queueing, forking, queue-running, and now file-locking mailman otherwise need to do.
Actually, i'm thinking about it from the administrators point of view. It's another, separate component to keep track of. If all the queuing, forking, queue-running, etc, are taken care of behind the scenes and in an airtight way, from the system manager's point of view it's simple as pie. A separate, independent component seems like a whole other matter. I suppose all the forking and queueing suggests having a clear division for a mail dispatching subsystem, i dunno. Lets see how others weigh in on the matter...
ken manheimer klm@python.org
participants (2)
-
Ken Manheimer -
The Dragon De Monsyne