Why are mails not being sent to all users (mails seem to disappear)?
Hello all,
we're using mailman-2.1.29 to operate a list where all users are moderated. The MTA used is postfix 3.2.0.
Generally everything seems to be working fine. I.e. when I send out an e-mail I have several confirmations that it arrives under the addresses subscribed to the list.
However, (at least) for one user the e-mail does not arrive. And I would at least like to find out the reason. But there's no trace of the address in any of the logfiles that should hold an entry (see below).
Generally I'm able to send messages using my MUA to the corresponding e-mail address using the very same postfix also delivering the mailman messages. So I'm guessing it's nothing like DMARC problems or the likes.
Like I said, what really puzzles me is that I cannot find the address in either
/var/log/mail (postfix)or /var/lib/mailman/logs/* (mailman)
/var/lib/mailman/logs/error doesn't contain any entries.
/var/lib/mailman/logs/bounce does contain some addresses with a bounce score of 1.0 which are for /other/ e-mail addresses also subscribed to the list in question.
/var/log/mail contains entries for /some/ of the addresses subscribed (including some bounces), but *not* for all.
/var/lib/mailman/logs/smtp gives me a log entry with the original messages and claims to have sent it to 416 recips (which is correct).
But overall I'm only able to find roughly half in either /var/log/mail or /var/lib/mailman/logs/*.
Why are not all 416 addresses listed in /var/log/mail, assuming that mailman handed 416 mails to postfix?
Do I have any chance of finding out what happened to the mails to the addresses which don't appear in any of the log files?
Any hints/insights are greatly appreciated!
Thanks -- Till
PRESENSE Technologies GmbH Sachsenstr. 5, D-20097 HH Geschäftsführer/Managing Directors AG Hamburg, HRB 107844 Till Dörges, Jürgen Sander USt-IdNr.: DE263765024
Wir sind wieder auf dem BSI IT-Sicherheitskongress
21.-23. Mai 2019 – Bonn
On 5/9/19 3:49 AM, Till Dörges wrote:
/var/lib/mailman/logs/smtp gives me a log entry with the original messages and claims to have sent it to 416 recips (which is correct).
I assume there's nothing in /var/lib/mailman/logs/smtp-failure since if there were, these would also be recorded as bounces.
But overall I'm only able to find roughly half in either /var/log/mail or /var/lib/mailman/logs/*.
Why are not all 416 addresses listed in /var/log/mail, assuming that mailman handed 416 mails to postfix?
Mailman did complete SMTP to Postfix for all 416 recipients. The /var/lib/mailman/logs/smtp log entry verifies that. What happened after that is a Postfix question.
Do I have any chance of finding out what happened to the mails to the addresses which don't appear in any of the log files?
I don't think this should happen without Postfix logging something, but if I have to guess, I'd guess you are not VERPing delivery to Postfix and the list is not personalized so Mailman is delivering to Postfix in transactions with up to SMTP_MAX_RCPTS (default 500) per transaction and Postfix's smtpd_recipient_limit is less.
I suggest you set SMTP_MAX_RCPTS = 10 or 20 in mm_cfg.py or better set VERP_DELIVERY_INTERVAL = 1 to enable VERP on all deliveries.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
thanks for your quick answer!
Am 09.05.19 um 14:31 schrieb Mark Sapiro:
I assume there's nothing in /var/lib/mailman/logs/smtp-failure since if there were, these would also be recorded as bounces.
/var/lib/mailman/logs/smtp-failure does not have any entries related to the issue.
But overall I'm only able to find roughly half in either /var/log/mail or /var/lib/mailman/logs/*. [...] Mailman did complete SMTP to Postfix for all 416 recipients. The /var/lib/mailman/logs/smtp log entry verifies that. What happened after that is a Postfix question.
Ack.
IIUC it seems in deed as if messages were lost between Mailman and Postfix, without any of them noticing/logging it.
The only indication I could find that there /might/ be problem was this warning with a suitable timestamp:
--- snip --- ... postfix/master...: warning: service ... has reached its process limit "10": new clients may experience noticeable delays --- snap ---
Do I have any chance of finding out what happened to the mails to the addresses which don't appear in any of the log files?
I don't think this should happen without Postfix logging something, but if I have to guess, I'd guess you are not VERPing delivery to Postfix and the list is not personalized so Mailman is delivering to Postfix in transactions with up to SMTP_MAX_RCPTS (default 500) per transaction and Postfix's smtpd_recipient_limit is less.
Postfix is configured with 1000:
smtpd_recipient_limit = 1000
Neither SMTP_MAX_RCPTS nor VERP_DELIVERY_INTERVAL was set explicitly, so I guess the defaults from /usr/lib/mailman/Mailman/Defaults.py are being used:
SMTP_MAX_RCPTS = 500 VERP_DELIVERY_INTERVAL = 0
I suggest you set SMTP_MAX_RCPTS = 10 or 20 in mm_cfg.py or better set VERP_DELIVERY_INTERVAL = 1 to enable VERP on all deliveries.
I will try with VERP_DELIVERY_INTERVAL set to 1, to see whether all messages are accounted for then.
IIRC, setting VERP_DELIVERY_INTERVAL = 1 effectively means to ignore SMTP_MAX_RCPTS and only submit 1 mail at a time to the MTA.
Curiously for the message that was not delivered to the specific user I did /not/ find a message by Postfix like the following in /var/log/mail:
--- snip --- ... to=<LISTADDR>, relay=local, delay=0.58, delays=0.01/0/0/0.57, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman post <LIST>) --- snap ---
To me that looks as if Postfix did in deed not complete sending all messages.
I will monitor the situation with these reduced rate settings applied.
Thanks again -- Till
Dipl.-Inform. Till Dörges doerges@pre-sense.de Tel. +49 - 40 - 244 2407 - 0 Fax +49 - 40 - 244 2407 - 24 PRESENSE Technologies GmbH Sachsenstr. 5, D-20097 HH Geschäftsführer/Managing Directors AG Hamburg, HRB 107844 Till Dörges, Jürgen Sander USt-IdNr.: DE263765024
Wir sind wieder auf dem BSI IT-Sicherheitskongress
21.-23. Mai 2019 – BonnOn 5/9/19 2:06 PM, Till Dörges wrote:
I will try with VERP_DELIVERY_INTERVAL set to 1, to see whether all messages are accounted for then.
IIRC, setting VERP_DELIVERY_INTERVAL = 1 effectively means to ignore SMTP_MAX_RCPTS and only submit 1 mail at a time to the MTA.
That's correct.
Curiously for the message that was not delivered to the specific user I did /not/ find a message by Postfix like the following in /var/log/mail:
--- snip --- ... to=<LISTADDR>, relay=local, delay=0.58, delays=0.01/0/0/0.57, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman post <LIST>) --- snap ---
To me that looks as if Postfix did in deed not complete sending all messages.
That message is delivery of the original incoming post to Mailman. That message should be logged at the time Mailman received the message. I.e., in Mailman's smtp log you see a time stamp and the smtp to list for 416 recipients in n.nnn seconds. The arrival time is a bit before the time stamp - n.nnn seconds.
I will monitor the situation with these reduced rate settings applied.
OK.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 09.05.19 um 23:33 schrieb Mark Sapiro:
On 5/9/19 2:06 PM, Till Dörges wrote:
Curiously for the message that was not delivered to the specific user I did /not/ find a message by Postfix like the following in /var/log/mail: [...] To me that looks as if Postfix did in deed not complete sending all messages. [...] I will monitor the situation with these reduced rate settings applied.
OK.
I figured it out. And it was neither a Mailman nor a Postfix problem.
systemd-journald simply enforced some rate limiting. After removing it (setting RateLimitBurst=0 in /etc/systemd/journald.conf), delivery for all mails can be seen in the logfiles.
Thanks again for your help, and sorry for the noise!
Regards -- Till
Dipl.-Inform. Till Dörges doerges@pre-sense.de PRESENSE Technologies GmbH Sachsenstr. 5, D-20097 HH Geschäftsführer/Managing Directors AG Hamburg, HRB 107844 Till Dörges, Jürgen Sander USt-IdNr.: DE263765024
On Thu, May 30, 2019 at 07:22:38PM +0200, Till Dörges wrote:
I figured it out. And it was neither a Mailman nor a Postfix problem.
systemd-journald simply enforced some rate limiting. After removing it (setting RateLimitBurst=0 in /etc/systemd/journald.conf), delivery for all mails can be seen in the logfiles.
Are you saying that the emails actually were sent, but systemd dropped the log messages so you wrongly thought they weren't sent?
Or that systemd prevented the emails from being sent in the first place, and didn't log the fact that it had done so?
I'm not sure which of those is more horrific, so I'm hoping I've misunderstood something because I will soon be migrating my mailman to a system using systemd.
-- Steven
On 5/30/19 7:46 PM, Steven D'Aprano wrote:
Or that systemd prevented the emails from being sent in the first place, and didn't log the fact that it had done so?
I don't think systemd metastasized its own smtpd just yet, so it probably can't do that. Fear not, though, it has its own DNS resolver already, I'm sure it won't take long.
-- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
Am 31.05.19 um 02:46 schrieb Steven D'Aprano:
On Thu, May 30, 2019 at 07:22:38PM +0200, Till Dörges wrote:
I figured it out. And it was neither a Mailman nor a Postfix problem.
systemd-journald simply enforced some rate limiting. After removing it (setting RateLimitBurst=0 in /etc/systemd/journald.conf), delivery for all mails can be seen in the logfiles.
Are you saying that the emails actually were sent, but systemd dropped the log messages so you wrongly thought they weren't sent?
That was the case. Only the log messages in syslog were missing.
To be clear: AFAICT everything was working as configured.
Regards -- Till
Dipl.-Inform. Till Dörges doerges@pre-sense.de PRESENSE Technologies GmbH Sachsenstr. 5, D-20097 HH Geschäftsführer/Managing Directors AG Hamburg, HRB 107844 Till Dörges, Jürgen Sander USt-IdNr.: DE263765024
participants (4)
-
Dimitri Maziuk -
Mark Sapiro -
Steven D'Aprano -
Till Dörges