![](https://secure.gravatar.com/avatar/310d3b2351983633dac3d7f25746bb3b.jpg?s=120&d=mm&r=g)
Did you ever find a solution to this issue?
I realize that this is a really old posting:
I send an eMail to one of the lists, I can see in my /var/log/maillog;
Jun 10 11:27:03 osiris sendmail[2756]: h5A9R392002756:
from=<Niklas.Nikitin at xxx
<http://mail.python.org/mailman/listinfo/mailman-users> >,
size=502, class=0, nrcpts=1,
msgid=<5.2.1.1.2.20030610112857.00b9b740 at xxx
<http://mail.python.org/mailman/listinfo/mailman-users> >,
proto=ESMTP, daemon=MTA,
relay=cassini.xxx [193.10.220.37]
Jun 10 11:27:03 osiris sendmail[2757]: h5A9R392002756:
to="|/usr/local/mailman/mail/mailman post nicke-test",
ctladdr=<nicke-test at xxx
<http://mail.python.org/mailman/listinfo/mailman-users> > (8/0),
delay=00:00:00, xdelay=00:00:00,
mailer=prog, pri=30704, dsn=2.0.0, stat=Sent
that the server is receiving the eMail and is forwarding it to
Mailman. But in Mailman's logs I will not find any information
that it has received the eMail and no eMail is sent to the members
on the list. If I dump all members of the list to a file, remove
the list, create the list again and add all members, the list is
working again for a while, and then it repeats again.
but I've run into a very similar problem with two mailing lists that are managed using the current release of Mailman.
The first symptom was that messages sent to the mailing lists were not distributed to the list, and did not show up in the list archive.
After my ISP reinstalled Mailman, messages to the list started to show up in the list archive again. But the messages were still not distributed to members of the list.
We found that if we add a new user, then a message to the list *is* delivered to that new user, but still not to any of the existing users.
If the *new* user posts a message, that message appears in the list archive and the message is delivered to the entire mailing list.
There doesn't seem to be a way to dump and reload the mailing list that preserves all per-list-member configuration options, so if possible, we'd like to avoid that workaround. Especially if it only works temporarily.
Any insight / help / pointers appreciated.
Beau
![](https://secure.gravatar.com/avatar/3b593e0746a5694a1d3f3cc533954645.jpg?s=120&d=mm&r=g)
On Thursday 26 April 2007 22:49:21 bjames@cisco.com wrote:
I had an oddity which may/maynot have some relevance..
We subscribed someone to a list with a specific email address. They confirmed the subscription using web interface. Subsequently mails from that member were received in the way you described and were passed to mailman but did not get posted to the list.
I suggest you examine the headers of incoming mails,as they are intially received, and outgoing mails to those subscribers who are not receiving the mails.. There could be some changes being made to the headers either by the subscribers server or your own that is causing mails to be not delivered.
david
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
bjames@cisco.com wrote:
today I have a similar (or the same?) problem. Running Mailman V2.1.9. Symptoms in my case: a new member (added with the command line utility add_members) is subscribed properly, the associated mail address gets a welcome message (so far so good), but mail to the mailing list arrives at Mailman, but is not distributed to the new user. I tried adding a second user, but the same problem.
I have checked a number of things from the FAQ to diagnose the problem but I see nothing unusual. No traces in the logs directory. And I'm sure Mailman has accepted the mail for further processing (I can see from the mail logs).
How can I trace these messages in Mailman? I looked in the qfiles/ subdirectories, but no files there... Is there any logging/debugging that can be turned on to see what steps Mailman takes to a) process the mail, (b) expand the mailing list and c) deliver the mail to the mail server. SMTP settings are default. Or is there any utility to quickly see what messages are in the Mailman queue for processing?
/rolf
![](https://secure.gravatar.com/avatar/334b870d5b26878a79b2dc4cfcc500bc.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld writes:
Have posts ever been delivered to any user? If not, the most likely Mailman-related problem is that due to your settings, the posts are being held. Have you checked the moderation interface? More likely IMO is that Mailman never received the posts because of failure to start Mailman's qrunners properly, or misconfiguration of the MTA.
On the contrary, if Mailman isn't writing to its logs and there's nothing in the qfiles directories, it seems very likely that Mailman has never received the messages.
Are there entries in logs/qrunner showing that all qrunners are working? Mine says:
Oct 18 17:14:29 2006 (15648) ArchRunner qrunner started.
Oct 18 17:14:30 2006 (15649) BounceRunner qrunner started.
Oct 18 17:14:33 2006 (15650) CommandRunner qrunner started.
Oct 18 17:14:35 2006 (15653) NewsRunner qrunner started.
Oct 18 17:14:35 2006 (15655) VirginRunner qrunner started.
Oct 18 17:14:35 2006 (15651) IncomingRunner qrunner started.
Oct 18 17:14:35 2006 (15654) OutgoingRunner qrunner started.
Oct 18 17:14:36 2006 (15656) RetryRunner qrunner started.
Does ps show that they are still running?
Fairly early in message processing, Mailman logs to either logs/post (showing that the post was accepted) or to logs/vette (indicating that the post was held or rejected, and why). When the post is delivered, there will be an entry in logs/smtp for each batch of deliveries (usually one per remote host, or one per user if personalization or VERP is being used). Are there really no logs at all? If there are logs, what do they say?
How can I trace these messages in Mailman? I looked in the qfiles/ subdirectories, but no files there...
Have you checked your MTA's queue? Based on your statements that there is nothing in the logs and nothing in the queues, I have to believe they're never getting to Mailman at all.
Is there any logging/debugging that can be turned on
No; the logs described plus the qfiles contain all the information that you should need for problems related to the delivery pipeline.
Or is there any utility to quickly see what messages are in the Mailman queue for processing?
To see what's there, I just use "ls -R /var/lib/mailman/qfiles" (YMMV depending on where Mailman is installed). To examine the qfiles themselves, use "cd /var/lib/mailman; bin/show_qfiles qfiles/in/*" (again, the cd directory will vary according to your installation).
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Stephen J. Turnbull wrote:
One minor point here. The post log entry is not written until the message is delivered by SMTPDirect.
Actually, if nothing goes wrong, there will be only one entry in the smtp log covering all deliveries. If things go wrong there will be perhaps multiple entries in both smtp and smtp-failure logs.
In spite of the minor corrections above, Stephen's advice is excellent.
To elaborate on Rolf's question about how Mailman processes a post. Assuming a standard configuration where the MTA delivers to the mail/mailman wrapper.
The MTA pipes the message to the wrapper which passes it to the scripts/post script which in turn places it in the qfiles/in queue. Nothing is logged in Mailman, but the MTA normally logs the delivery.
IncomingRunner picks up the queue entry from qfiles/in and passes it through a pipeline of handlers (GLOBAL_PIPELINE - defined in Defaults.py, possibly modified in mm_cfg.py - unless the list defines a pipeline attribute to override it). The initial handlers check header_filter_rules, look for an Approved: header and possibly generate an autoresponse. The next set of handlers checks for moderation and other holds and does content filtering, emergency moderation and topic flagging. Any of these handlers can raise exceptions to request IncomingRunner to discard, reject or hold the message at that point. Discards and holds are logged in the vette log. Rejects are not logged, but result in a reject message to the poster. Any other exceptions cause the message to be moved to qfiles/shunt and the exception is logged in the error log.
The next set of handlers determines the recipient addresses and possibly removes and/or modifies some message headers.
Then handlers add the message to lists/<listname>/digest.mbox if the list is digestable (and possibly trigger a digest on size), add the message to qfiles/archive for processing by ArchRunner, possibly add the message to qfiles/news for processing by NewsRunner, update the list's last_post_time, send an acknowledgement to the poster if requested, and add the message to qfiles/out for processing by OutgoingRunner.
- ArchRunner picks up the queue entry from qfiles/archive and adds it the the list's archive.
4)NewsRunner picks up the queue entry from qfiles/news and delivers it to Usenet.
5)OutgoingRunner picks up the queue entry from qfiles/out and calls SMTPDirect to deliver it to the outgoing MTA via SMTP. SMTPDirect logs posts to the post log and logs a bit more detail to the smtp log. Any SMTP failures are logged to the smtp-failure log and treated as bounces or queued for retry as appropriate.
- Any unanticipated exceptions in any of the runners cause the message to be moved to qfiles/shunt and the exception to be logged in the error log.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Hello, Mark,
Mark Sapiro wrote:
OK, thus only after the message has been delivered to at least one subscriber (or the first MTA in the chain of delivery to at least one subscriber).
In my case there were no entries in smtp nor in smtp-failure.
In spite of the minor corrections above, Stephen's advice is excellent.
To elaborate on Rolf's question about how Mailman processes a post.
Thanks _very much_ for the detailed explanation of the inner working of Mailman. I really appreciate your detailed description!
Now here's the problem, I think. The MTA logs the message as being delivered to Mailman (via the wrapper script $MAILMANDIR/mailman/bin/mailman), so from the MTA's point of view it's done. Now, as the sent message cannot be found under qfiles, and as none of the $MAILMANDIR/logs/* files is modified, it seems as if the message disappeared in a black hole. Is there no way to enable debugging in the wrapper script? It would show the first action of Mailman, I assume.
FYI I made sure that the recipients where no digest users.
/rolf
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld wrote:
Yes, the first MTA, namely the one on localhost, port 25 unless overridden by SMTPHOST/SMTPPORT in mm_cfg.py. There is no guarantee this MTA has sent the message further. You need to refer the the MTA's logs for that.
In my case there were no entries in smtp nor in smtp-failure.
So the message didn't get that far.
There are at least two possibilities here.
The post was rejected (not held or discarded). A reject is not logged, but there should be an associated entry in Mailman's smtp log with a mailman generated message-id and 1 recipient for the reject message sent to the poster.
The wrapper is for a different Mailman installation with a different qfiles directory and probably no qrunners running and probably different log files too.
The wrapper itself (I would expect it to be located at $MAILMANDIR/mail/mailman, not $MAILMANDIR/mailman/bin/mailman) has the path to the scripts/ directory compiled in and doesn't have any debug switches. It will invoke the scripts/post script and presumably this happens OK or it will return a failure status to the MTA which would be logged by the MTA.
You can add debugging to the scripts/post script. If you look at this script, you will see that it already detects a missing or invalid list name from the wrapper and writes to stderr which is logged to both the post and error logs.
You could add some more output, say by adding the last two lines of
inq = get_switchboard(mm_cfg.INQUEUE_DIR)
inq.enqueue(sys.stdin.read(),
listname=listname,
tolist=1, _plaintext=1)
print >> sys.stderr, 'Post for %s queued in %s' \
% (listname, mm_cfg.INQUEUE_DIR)
If you don't see this in the post and error logs, you are not looking at the right installation or (third possibility) the right log files. Look in mm_cfg.py/Defaults.py for the definition of LOG_DIR and anything else used in its definition.
This can be tricky. LOG_DIR is defined in Defaults.py in terms of VAR_PREFIX. If VAR_PREFIX is redefined in mm_cfg.py, this will not redefine LOG_DIR as LOG_DIR was already defined in Defaults.py in terms of the Defaults.py definition of VAR_PREFIX.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/395215a635f89ee4fd6d9dfe8453afae.jpg?s=120&d=mm&r=g)
Mark Sapiro writes:
2.1.9's Defaults.py has the following warning, which applies to both LOG_DIR and VAR_PREFIX. So if either appears in mm_cfg.py, there are legitimate grounds for a harsh complaint to the vendor or installer.
##### # Nothing below here is user configurable. Most of these values are in this # file for internal system convenience. Don't change any of them or override # any of them in your mm_cfg.py file! #####
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Hi,
[I changed the name of this thread, as I realized that I more or less hijacked this thread from bjames@cisco.com and two threads with the same subject proceeded in parallel, my sincere apologies to bjames@cisco.com and to the list]
Mark Sapiro wrote:
OK, there was no message submission to the MTA, and this corresponds to the fact that there are no entries in smtp and smtp-failure logfiles.
There is none.
I see no other installation; I have only one in /usr/local/mailman.
The wrapper itself (I would expect it to be located at $MAILMANDIR/mail/mailman, not $MAILMANDIR/mailman/bin/mailman)
correct, I made a wrong assumption when I was typing. I doublechecked and the mailman wrapper is at $MAILMANDIR/mail/mailman.
The logfile within the MTA shows:
11:26:48.08: middleman version 0.2 starting, pid 25427 11:26:48.16: channel: mailman 11:26:48.17: option file: IMTA_TABLE:mailman_option 11:26:48.17: Opening the channel option file 11:26:48.17: mtaOptionStart: succeeded 11:26:48.17: WRAPPER=/usr/local/mailman/mail/mailman 11:26:48.17: MAX_THREADS=5 11:26:48.17: mtaDequeueStart() 11:26:48.17: #1: process_message() callback 11:26:48.17: #1: build_operations() starting 11:26:48.17: #1: first address: allloc5estatic-owner@lists.domain.com 11:26:48.17: #1: localpart: allloc5estatic-owner 11:26:48.17: #1: listname: allloc5estatic 11:26:48.17: #1: command: owner 11:26:48.17: #1: build_operations() returning 11:26:48.17: #1: get_messagefile() starting 11:26:48.17: #1: opening tmpfile 11:26:48.19: #1: wrote tmpfile successfully 11:26:48.19: #1: get_messagefile() returning 11:26:48.19: #1: running wrapper for every recipient 11:26:48.19: #1: seeking back to the beginning of the message 11:26:48.19: #1: about to fork() 11:26:48.19: #1: command completed successfully 11:26:48.19: #1: cleaning up 11:26:48.19: #1: mm_opers_free() starting 11:26:48.19: #1: mm_opers_free() returning 11:26:48.19: #1: waiting child 25428 11:26:48.20: #1: dequeue finished 11:26:48.21: #1: command completed successfully 11:26:48.21: #1: cleaning up 11:26:48.21: #1: mm_opers_free() starting 11:26:48.21: #1: mm_opers_free() returning 11:26:48.21: #1: dequeue finished 11:29:48.00: mtaDequeueStart() returned
I have added the last two lines, but nothing is written to the post logfile.
The mm_cfg.py does not define LOG_DIR. In /usr/local/mailman/Mailman/Defaults.py it is defined as:
LOG_DIR = os.path.join(VAR_PREFIX, 'logs')
where VAR_PREFIX is defined in the same file as:
VAR_PREFIX = '/usr/local/mailman'
I see; however in my case this seems to be not the problem, as mm_cfg.py does not define LOG_DIR nor VAR_PREFIX.
Now, I tested the following:
I became user mailman mailman> cd /usr/local/mailman/scripts mailman> ./post <listname> <message input> <ctrl-D>
and indeed something showed up in the locks logfile, as well as in the post logfile and in the bounce logfile (as the sender address was not defined, mailman could not authorize the message submissions, which ended in a bounce). So it seems that the calling program within the MTA gets a positive success status from mailman, but something is wrong during message submission. To repeat part of he logfile:
[...] 11:26:48.19: #1: running wrapper for every recipient 11:26:48.19: #1: seeking back to the beginning of the message 11:26:48.19: #1: about to fork() 11:26:48.19: #1: command completed successfully 11:26:48.19: #1: cleaning up 11:26:48.19: #1: mm_opers_free() starting 11:26:48.19: #1: mm_opers_free() returning 11:26:48.19: #1: waiting child 25428 11:26:48.20: #1: dequeue finished 11:26:48.21: #1: command completed successfully 11:26:48.21: #1: cleaning up [...]
/rolf
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld wrote:
I see two possibilities.
the wrapper is failing and this fact is not communicated back to the MTA. Possibly there is a group mismatch or some other problem, but these result in stderr output from the wrapper and a non-zero exit status, both of which should be seen by the MTA.
the wrapper's precompiled path to the scripts directory is to something other than /usr/local/mailman/scripts.
It is not clear to me from your initial posts whether this issue involves just one list or all lists and whether things this list (or all lists) ever worked.
In any case, you could try your test similarly to the above, but using the wrapper as in
cd /usr/local/mailman/mail (or whatever the correct directory is) ./mailman post <listname> <message input> <ctrl-D>
The first attempt will probably result in a group mismatch error before it reads any message input. This will at least tell you what group it is expecting which may help if there is really a group mismatch with the MTA.
Then if you can try it again as the expected group, you can see if it actually posts the message.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Mark Sapiro wrote:
It involves all created lists. And the odd thing is that it seems that the lists sometimes work, and sometimes not. After more investigation I found the following pattern: when I create a new list and add one or more members to it (using add_members), and I post a message to this list using one of the subscribed addresses as the sender address, this posting disappears (as described in my previous messages). However, when I add new members (so a second add_members action), and when I send mail after that memberlist change, then the message is delivered properly!
Thanks for this troubleshooting suggestion. Is there a way I can specify a sender address for this wrapper?
/rolf
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld wrote:
That is very odd. Since the problem appears to be in the interaction between the MTA and the wrapper, and the wrapper/post script do not look at the contents of a post or really any other attributes of it - they just queue it for IncomingRunner. The wrapper doesn't even look at the list name; it just passes it to the post script. Also, the act of adding more members to a list does not affect anything that the MTA looks at (I think, see next paragraph).
The log you posted previously seems to say the MTA delivering to Mailman is 'middleman'. I don't know anything at all about this MTA. Another hint in a prior post tells me it may have some programatic way (rather than aliases) for determining whether and how a post should be delivered to Mailman. How does this work and what does it look at?
What user:group does the MTA use to invoke the wrapper? Can you invoke the wrapper manually as below as that user:group and see if it reports a group mismatch error
If you mean an envelope sender, include a "From " line as the first line of the message header. If you mean just the sender for list membership tests, the regular From: header should be sufficient.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/6eb1fb4bc1be82fffe9e6a7288f8fb22.jpg?s=120&d=mm&r=g)
Mark Sapiro wrote:
I hop into this thread as this is of interest to me, too.
Middleman is a channel daemon for Sun Java System Messaging Server (you may know it as iPlanet Messaging Server or PMDF). I wrote middleman to integrate Mailman with SJSMS as there was now thorough solution available. It's still a work in progress, but works for us at the University of Turku.
Now, there was a bug in middleman and I've asked Rolf to send me some debugging info to find where the problem is. Until then there's nothing to do but speculate. But I doubt the problem is in Mailman, although Rolf's description of the problem seems very odd, I must say.
If anyone is interested in the integration, you may ask me off list if you wish.
-- Eino Tuominen
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Eino Tuominen wrote:
the patched version of middleman solved the mailman problems I reported. It runs fine now. Many thanks for the help of the list, especially the help of Mark Sapiro. My understanding of Mailman has much improved!
If anyone is interested in the integration, you may ask me off list if you wish.
I wonder if there's an interest in documenting the integration of mailman with Sun Java System Messaging Server, as in chapter 6 of the Gnu Mailman installation manual (http://www.gnu.org/software/mailman/mailman-install/index.html)? I'm volunteering, but on the other hand, the README file written by Eino for the middleman interface code describes the integration process very well.
/rolf
![](https://secure.gravatar.com/avatar/3b593e0746a5694a1d3f3cc533954645.jpg?s=120&d=mm&r=g)
On Thursday 26 April 2007 22:49:21 bjames@cisco.com wrote:
I had an oddity which may/maynot have some relevance..
We subscribed someone to a list with a specific email address. They confirmed the subscription using web interface. Subsequently mails from that member were received in the way you described and were passed to mailman but did not get posted to the list.
I suggest you examine the headers of incoming mails,as they are intially received, and outgoing mails to those subscribers who are not receiving the mails.. There could be some changes being made to the headers either by the subscribers server or your own that is causing mails to be not delivered.
david
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
bjames@cisco.com wrote:
today I have a similar (or the same?) problem. Running Mailman V2.1.9. Symptoms in my case: a new member (added with the command line utility add_members) is subscribed properly, the associated mail address gets a welcome message (so far so good), but mail to the mailing list arrives at Mailman, but is not distributed to the new user. I tried adding a second user, but the same problem.
I have checked a number of things from the FAQ to diagnose the problem but I see nothing unusual. No traces in the logs directory. And I'm sure Mailman has accepted the mail for further processing (I can see from the mail logs).
How can I trace these messages in Mailman? I looked in the qfiles/ subdirectories, but no files there... Is there any logging/debugging that can be turned on to see what steps Mailman takes to a) process the mail, (b) expand the mailing list and c) deliver the mail to the mail server. SMTP settings are default. Or is there any utility to quickly see what messages are in the Mailman queue for processing?
/rolf
![](https://secure.gravatar.com/avatar/334b870d5b26878a79b2dc4cfcc500bc.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld writes:
Have posts ever been delivered to any user? If not, the most likely Mailman-related problem is that due to your settings, the posts are being held. Have you checked the moderation interface? More likely IMO is that Mailman never received the posts because of failure to start Mailman's qrunners properly, or misconfiguration of the MTA.
On the contrary, if Mailman isn't writing to its logs and there's nothing in the qfiles directories, it seems very likely that Mailman has never received the messages.
Are there entries in logs/qrunner showing that all qrunners are working? Mine says:
Oct 18 17:14:29 2006 (15648) ArchRunner qrunner started.
Oct 18 17:14:30 2006 (15649) BounceRunner qrunner started.
Oct 18 17:14:33 2006 (15650) CommandRunner qrunner started.
Oct 18 17:14:35 2006 (15653) NewsRunner qrunner started.
Oct 18 17:14:35 2006 (15655) VirginRunner qrunner started.
Oct 18 17:14:35 2006 (15651) IncomingRunner qrunner started.
Oct 18 17:14:35 2006 (15654) OutgoingRunner qrunner started.
Oct 18 17:14:36 2006 (15656) RetryRunner qrunner started.
Does ps show that they are still running?
Fairly early in message processing, Mailman logs to either logs/post (showing that the post was accepted) or to logs/vette (indicating that the post was held or rejected, and why). When the post is delivered, there will be an entry in logs/smtp for each batch of deliveries (usually one per remote host, or one per user if personalization or VERP is being used). Are there really no logs at all? If there are logs, what do they say?
How can I trace these messages in Mailman? I looked in the qfiles/ subdirectories, but no files there...
Have you checked your MTA's queue? Based on your statements that there is nothing in the logs and nothing in the queues, I have to believe they're never getting to Mailman at all.
Is there any logging/debugging that can be turned on
No; the logs described plus the qfiles contain all the information that you should need for problems related to the delivery pipeline.
Or is there any utility to quickly see what messages are in the Mailman queue for processing?
To see what's there, I just use "ls -R /var/lib/mailman/qfiles" (YMMV depending on where Mailman is installed). To examine the qfiles themselves, use "cd /var/lib/mailman; bin/show_qfiles qfiles/in/*" (again, the cd directory will vary according to your installation).
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Stephen J. Turnbull wrote:
One minor point here. The post log entry is not written until the message is delivered by SMTPDirect.
Actually, if nothing goes wrong, there will be only one entry in the smtp log covering all deliveries. If things go wrong there will be perhaps multiple entries in both smtp and smtp-failure logs.
In spite of the minor corrections above, Stephen's advice is excellent.
To elaborate on Rolf's question about how Mailman processes a post. Assuming a standard configuration where the MTA delivers to the mail/mailman wrapper.
The MTA pipes the message to the wrapper which passes it to the scripts/post script which in turn places it in the qfiles/in queue. Nothing is logged in Mailman, but the MTA normally logs the delivery.
IncomingRunner picks up the queue entry from qfiles/in and passes it through a pipeline of handlers (GLOBAL_PIPELINE - defined in Defaults.py, possibly modified in mm_cfg.py - unless the list defines a pipeline attribute to override it). The initial handlers check header_filter_rules, look for an Approved: header and possibly generate an autoresponse. The next set of handlers checks for moderation and other holds and does content filtering, emergency moderation and topic flagging. Any of these handlers can raise exceptions to request IncomingRunner to discard, reject or hold the message at that point. Discards and holds are logged in the vette log. Rejects are not logged, but result in a reject message to the poster. Any other exceptions cause the message to be moved to qfiles/shunt and the exception is logged in the error log.
The next set of handlers determines the recipient addresses and possibly removes and/or modifies some message headers.
Then handlers add the message to lists/<listname>/digest.mbox if the list is digestable (and possibly trigger a digest on size), add the message to qfiles/archive for processing by ArchRunner, possibly add the message to qfiles/news for processing by NewsRunner, update the list's last_post_time, send an acknowledgement to the poster if requested, and add the message to qfiles/out for processing by OutgoingRunner.
- ArchRunner picks up the queue entry from qfiles/archive and adds it the the list's archive.
4)NewsRunner picks up the queue entry from qfiles/news and delivers it to Usenet.
5)OutgoingRunner picks up the queue entry from qfiles/out and calls SMTPDirect to deliver it to the outgoing MTA via SMTP. SMTPDirect logs posts to the post log and logs a bit more detail to the smtp log. Any SMTP failures are logged to the smtp-failure log and treated as bounces or queued for retry as appropriate.
- Any unanticipated exceptions in any of the runners cause the message to be moved to qfiles/shunt and the exception to be logged in the error log.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Hello, Mark,
Mark Sapiro wrote:
OK, thus only after the message has been delivered to at least one subscriber (or the first MTA in the chain of delivery to at least one subscriber).
In my case there were no entries in smtp nor in smtp-failure.
In spite of the minor corrections above, Stephen's advice is excellent.
To elaborate on Rolf's question about how Mailman processes a post.
Thanks _very much_ for the detailed explanation of the inner working of Mailman. I really appreciate your detailed description!
Now here's the problem, I think. The MTA logs the message as being delivered to Mailman (via the wrapper script $MAILMANDIR/mailman/bin/mailman), so from the MTA's point of view it's done. Now, as the sent message cannot be found under qfiles, and as none of the $MAILMANDIR/logs/* files is modified, it seems as if the message disappeared in a black hole. Is there no way to enable debugging in the wrapper script? It would show the first action of Mailman, I assume.
FYI I made sure that the recipients where no digest users.
/rolf
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld wrote:
Yes, the first MTA, namely the one on localhost, port 25 unless overridden by SMTPHOST/SMTPPORT in mm_cfg.py. There is no guarantee this MTA has sent the message further. You need to refer the the MTA's logs for that.
In my case there were no entries in smtp nor in smtp-failure.
So the message didn't get that far.
There are at least two possibilities here.
The post was rejected (not held or discarded). A reject is not logged, but there should be an associated entry in Mailman's smtp log with a mailman generated message-id and 1 recipient for the reject message sent to the poster.
The wrapper is for a different Mailman installation with a different qfiles directory and probably no qrunners running and probably different log files too.
The wrapper itself (I would expect it to be located at $MAILMANDIR/mail/mailman, not $MAILMANDIR/mailman/bin/mailman) has the path to the scripts/ directory compiled in and doesn't have any debug switches. It will invoke the scripts/post script and presumably this happens OK or it will return a failure status to the MTA which would be logged by the MTA.
You can add debugging to the scripts/post script. If you look at this script, you will see that it already detects a missing or invalid list name from the wrapper and writes to stderr which is logged to both the post and error logs.
You could add some more output, say by adding the last two lines of
inq = get_switchboard(mm_cfg.INQUEUE_DIR)
inq.enqueue(sys.stdin.read(),
listname=listname,
tolist=1, _plaintext=1)
print >> sys.stderr, 'Post for %s queued in %s' \
% (listname, mm_cfg.INQUEUE_DIR)
If you don't see this in the post and error logs, you are not looking at the right installation or (third possibility) the right log files. Look in mm_cfg.py/Defaults.py for the definition of LOG_DIR and anything else used in its definition.
This can be tricky. LOG_DIR is defined in Defaults.py in terms of VAR_PREFIX. If VAR_PREFIX is redefined in mm_cfg.py, this will not redefine LOG_DIR as LOG_DIR was already defined in Defaults.py in terms of the Defaults.py definition of VAR_PREFIX.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/395215a635f89ee4fd6d9dfe8453afae.jpg?s=120&d=mm&r=g)
Mark Sapiro writes:
2.1.9's Defaults.py has the following warning, which applies to both LOG_DIR and VAR_PREFIX. So if either appears in mm_cfg.py, there are legitimate grounds for a harsh complaint to the vendor or installer.
##### # Nothing below here is user configurable. Most of these values are in this # file for internal system convenience. Don't change any of them or override # any of them in your mm_cfg.py file! #####
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Hi,
[I changed the name of this thread, as I realized that I more or less hijacked this thread from bjames@cisco.com and two threads with the same subject proceeded in parallel, my sincere apologies to bjames@cisco.com and to the list]
Mark Sapiro wrote:
OK, there was no message submission to the MTA, and this corresponds to the fact that there are no entries in smtp and smtp-failure logfiles.
There is none.
I see no other installation; I have only one in /usr/local/mailman.
The wrapper itself (I would expect it to be located at $MAILMANDIR/mail/mailman, not $MAILMANDIR/mailman/bin/mailman)
correct, I made a wrong assumption when I was typing. I doublechecked and the mailman wrapper is at $MAILMANDIR/mail/mailman.
The logfile within the MTA shows:
11:26:48.08: middleman version 0.2 starting, pid 25427 11:26:48.16: channel: mailman 11:26:48.17: option file: IMTA_TABLE:mailman_option 11:26:48.17: Opening the channel option file 11:26:48.17: mtaOptionStart: succeeded 11:26:48.17: WRAPPER=/usr/local/mailman/mail/mailman 11:26:48.17: MAX_THREADS=5 11:26:48.17: mtaDequeueStart() 11:26:48.17: #1: process_message() callback 11:26:48.17: #1: build_operations() starting 11:26:48.17: #1: first address: allloc5estatic-owner@lists.domain.com 11:26:48.17: #1: localpart: allloc5estatic-owner 11:26:48.17: #1: listname: allloc5estatic 11:26:48.17: #1: command: owner 11:26:48.17: #1: build_operations() returning 11:26:48.17: #1: get_messagefile() starting 11:26:48.17: #1: opening tmpfile 11:26:48.19: #1: wrote tmpfile successfully 11:26:48.19: #1: get_messagefile() returning 11:26:48.19: #1: running wrapper for every recipient 11:26:48.19: #1: seeking back to the beginning of the message 11:26:48.19: #1: about to fork() 11:26:48.19: #1: command completed successfully 11:26:48.19: #1: cleaning up 11:26:48.19: #1: mm_opers_free() starting 11:26:48.19: #1: mm_opers_free() returning 11:26:48.19: #1: waiting child 25428 11:26:48.20: #1: dequeue finished 11:26:48.21: #1: command completed successfully 11:26:48.21: #1: cleaning up 11:26:48.21: #1: mm_opers_free() starting 11:26:48.21: #1: mm_opers_free() returning 11:26:48.21: #1: dequeue finished 11:29:48.00: mtaDequeueStart() returned
I have added the last two lines, but nothing is written to the post logfile.
The mm_cfg.py does not define LOG_DIR. In /usr/local/mailman/Mailman/Defaults.py it is defined as:
LOG_DIR = os.path.join(VAR_PREFIX, 'logs')
where VAR_PREFIX is defined in the same file as:
VAR_PREFIX = '/usr/local/mailman'
I see; however in my case this seems to be not the problem, as mm_cfg.py does not define LOG_DIR nor VAR_PREFIX.
Now, I tested the following:
I became user mailman mailman> cd /usr/local/mailman/scripts mailman> ./post <listname> <message input> <ctrl-D>
and indeed something showed up in the locks logfile, as well as in the post logfile and in the bounce logfile (as the sender address was not defined, mailman could not authorize the message submissions, which ended in a bounce). So it seems that the calling program within the MTA gets a positive success status from mailman, but something is wrong during message submission. To repeat part of he logfile:
[...] 11:26:48.19: #1: running wrapper for every recipient 11:26:48.19: #1: seeking back to the beginning of the message 11:26:48.19: #1: about to fork() 11:26:48.19: #1: command completed successfully 11:26:48.19: #1: cleaning up 11:26:48.19: #1: mm_opers_free() starting 11:26:48.19: #1: mm_opers_free() returning 11:26:48.19: #1: waiting child 25428 11:26:48.20: #1: dequeue finished 11:26:48.21: #1: command completed successfully 11:26:48.21: #1: cleaning up [...]
/rolf
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld wrote:
I see two possibilities.
the wrapper is failing and this fact is not communicated back to the MTA. Possibly there is a group mismatch or some other problem, but these result in stderr output from the wrapper and a non-zero exit status, both of which should be seen by the MTA.
the wrapper's precompiled path to the scripts directory is to something other than /usr/local/mailman/scripts.
It is not clear to me from your initial posts whether this issue involves just one list or all lists and whether things this list (or all lists) ever worked.
In any case, you could try your test similarly to the above, but using the wrapper as in
cd /usr/local/mailman/mail (or whatever the correct directory is) ./mailman post <listname> <message input> <ctrl-D>
The first attempt will probably result in a group mismatch error before it reads any message input. This will at least tell you what group it is expecting which may help if there is really a group mismatch with the MTA.
Then if you can try it again as the expected group, you can see if it actually posts the message.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Mark Sapiro wrote:
It involves all created lists. And the odd thing is that it seems that the lists sometimes work, and sometimes not. After more investigation I found the following pattern: when I create a new list and add one or more members to it (using add_members), and I post a message to this list using one of the subscribed addresses as the sender address, this posting disappears (as described in my previous messages). However, when I add new members (so a second add_members action), and when I send mail after that memberlist change, then the message is delivered properly!
Thanks for this troubleshooting suggestion. Is there a way I can specify a sender address for this wrapper?
/rolf
![](https://secure.gravatar.com/avatar/746f7519ba02fb0d815e59f305c53fa2.jpg?s=120&d=mm&r=g)
Rolf E. Sonneveld wrote:
That is very odd. Since the problem appears to be in the interaction between the MTA and the wrapper, and the wrapper/post script do not look at the contents of a post or really any other attributes of it - they just queue it for IncomingRunner. The wrapper doesn't even look at the list name; it just passes it to the post script. Also, the act of adding more members to a list does not affect anything that the MTA looks at (I think, see next paragraph).
The log you posted previously seems to say the MTA delivering to Mailman is 'middleman'. I don't know anything at all about this MTA. Another hint in a prior post tells me it may have some programatic way (rather than aliases) for determining whether and how a post should be delivered to Mailman. How does this work and what does it look at?
What user:group does the MTA use to invoke the wrapper? Can you invoke the wrapper manually as below as that user:group and see if it reports a group mismatch error
If you mean an envelope sender, include a "From " line as the first line of the message header. If you mean just the sender for list membership tests, the regular From: header should be sufficient.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/6eb1fb4bc1be82fffe9e6a7288f8fb22.jpg?s=120&d=mm&r=g)
Mark Sapiro wrote:
I hop into this thread as this is of interest to me, too.
Middleman is a channel daemon for Sun Java System Messaging Server (you may know it as iPlanet Messaging Server or PMDF). I wrote middleman to integrate Mailman with SJSMS as there was now thorough solution available. It's still a work in progress, but works for us at the University of Turku.
Now, there was a bug in middleman and I've asked Rolf to send me some debugging info to find where the problem is. Until then there's nothing to do but speculate. But I doubt the problem is in Mailman, although Rolf's description of the problem seems very odd, I must say.
If anyone is interested in the integration, you may ask me off list if you wish.
-- Eino Tuominen
![](https://secure.gravatar.com/avatar/04ed7790224245da8f655597c3487b65.jpg?s=120&d=mm&r=g)
Eino Tuominen wrote:
the patched version of middleman solved the mailman problems I reported. It runs fine now. Many thanks for the help of the list, especially the help of Mark Sapiro. My understanding of Mailman has much improved!
If anyone is interested in the integration, you may ask me off list if you wish.
I wonder if there's an interest in documenting the integration of mailman with Sun Java System Messaging Server, as in chapter 6 of the Gnu Mailman installation manual (http://www.gnu.org/software/mailman/mailman-install/index.html)? I'm volunteering, but on the other hand, the README file written by Eino for the middleman interface code describes the integration process very well.
/rolf
participants (7)
-
bjames@cisco.com
-
David Southwell
-
Eino Tuominen
-
Mark Sapiro
-
Rolf E. Sonneveld
-
Stephen J. Turnbull
-
Stephen J. Turnbull