![](https://secure.gravatar.com/avatar/51b96f7367cfd681834dd636ad1c25d8.jpg?s=120&d=mm&r=g)
Hi all,
Recently I found a fairly huge amount of files in the shunt queue (around 1500). All I could gather from the error log is that most of them were from an old bug in parsing weird charsets and the later ones were due to the post and smtp-failur logfiles being owned by root. So I fixed the perms (had already upgraded to 2.1.8 a month or so ago) and all was well.
Then I ran bin/unshunt.
Then people started complaining that they were getting a few hundred old messages. :)
What am I missing here? From what I've been able to dig up on the list archives the qfiles/shunt directory should only contain messages that never made it out to any lists due to some error in config, perms or some bug in mailman itself. But it's looking like these all did somehow make it through mailman previously and were sent a second time. A few other tidbits:
-looking at some samples, msgids are the same, so it wasn't a user getting a rejection and then posting again
-the dupes DO NOT show up in the web archive
Any other info I can supply here? Mailman 2.1.8, ruby 1.8.2, FreeBSD 4.7.
Thanks,
Charles
Charles Sprickman NetEng/SysAdmin Bway.net - New York's Best Internet - www.bway.net spork@bway.net - 212.655.9344
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Charles Sprickman wrote:
There are several possibilities here.
users are complaining that they received messages posted long ago in threads long since dead, not necessarily that they previously received the same message (message-id).
when the original didn't reach the list, the OP 'fixed' the problem and reposted, thus the current message only appears to be a dupe.
if a message is shunted because SMTPDirect.py can't write the 'post' log, it has already been sent, but OutgoingRunner.py doesn't know this when it catches the exception and shunts the message. So those messages which were shunted because of ownership/permission issues on the 'post' log probably were duplicated when you ran unshunt
This is not always true, as in case 3) above. Thus, you have to use judgement in unshunting.
OK, so that rules out 2).
-the dupes DO NOT show up in the web archive
Archiving is a separate process. Depending on where in processing the error occurred, lots of possibilities ensue.
If the error occurs early in IncomingRunner, the post will be neither archived nor delivered until the problem is fixed and the message is unshunted.
In the case where the error occurs in delivery under OutgoingRunner, the message will have been archived, and when unshunted will be processed only by OutgoingRunner and not by ArchRunner, so no duplicate archive entry results. But, in this case, it is possible that some or all receipients received the original delivery and will receive duplicates when the message is unshunted.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/51b96f7367cfd681834dd636ad1c25d8.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006, Mark Sapiro wrote:
I think that solves the mystery then.
Will do. Also set up an snmp monitoring process to watch the shunt queue and let me know when things are piling up so I can investigate.
Excellent info. I put a question regarding this in the FAQ, so now I'll answer it.
Thanks for the insight Mark, very much appreciated.
Charles
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Charles Sprickman wrote:
There are several possibilities here.
users are complaining that they received messages posted long ago in threads long since dead, not necessarily that they previously received the same message (message-id).
when the original didn't reach the list, the OP 'fixed' the problem and reposted, thus the current message only appears to be a dupe.
if a message is shunted because SMTPDirect.py can't write the 'post' log, it has already been sent, but OutgoingRunner.py doesn't know this when it catches the exception and shunts the message. So those messages which were shunted because of ownership/permission issues on the 'post' log probably were duplicated when you ran unshunt
This is not always true, as in case 3) above. Thus, you have to use judgement in unshunting.
OK, so that rules out 2).
-the dupes DO NOT show up in the web archive
Archiving is a separate process. Depending on where in processing the error occurred, lots of possibilities ensue.
If the error occurs early in IncomingRunner, the post will be neither archived nor delivered until the problem is fixed and the message is unshunted.
In the case where the error occurs in delivery under OutgoingRunner, the message will have been archived, and when unshunted will be processed only by OutgoingRunner and not by ArchRunner, so no duplicate archive entry results. But, in this case, it is possible that some or all receipients received the original delivery and will receive duplicates when the message is unshunted.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/51b96f7367cfd681834dd636ad1c25d8.jpg?s=120&d=mm&r=g)
On Mon, 31 Jul 2006, Mark Sapiro wrote:
I think that solves the mystery then.
Will do. Also set up an snmp monitoring process to watch the shunt queue and let me know when things are piling up so I can investigate.
Excellent info. I put a question regarding this in the FAQ, so now I'll answer it.
Thanks for the insight Mark, very much appreciated.
Charles
participants (2)
-
Charles Sprickman
-
Mark Sapiro