Hello
How do I speed up mailman so that it works instantaneously and stops storing messages.
Ruben
So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
It's a factor of postfix than it is for mailman. However you could reduce the level of moderation so things can pass scrutiny as fast on the mailman side.
Cheers!
Sent from my LG G4 Kindly excuse brevity and typos On 28 Feb 2016 07:58, "Ruben Safir" mrbrklyn@panix.com wrote:
Hello
How do I speed up mailman so that it works instantaneously and stops storing messages.
Ruben
So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/seun.ojedeji%40gmail.c...
On 02/28/2016 03:15 AM, Seun Ojedeji wrote:
It's a factor of postfix than it is for mailman. However you could reduce the level of moderation so things can pass scrutiny as fast on the mailman side.
everything directly mailed through postfix goes straight through without any delay. So I'm at a loss to understand it.
Cheers!
Sent from my LG G4 Kindly excuse brevity and typos On 28 Feb 2016 07:58, "Ruben Safir" mrbrklyn@panix.com wrote:
Hello
How do I speed up mailman so that it works instantaneously and stops storing messages.
Ruben
So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/seun.ojedeji%40gmail.c...
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Ruben Safir writes:
How do I speed up mailman so that it works instantaneously and stops storing messages.
If you mean "Mailman is administratively holding messages until released by a moderator," you need to tell us more about your current configuration and needs. There are several reasons why Mailman will hold messages -- what is the reason that Mailman gives for holding your messages?
OTOH, if it's performance problem and messages are piling up in queues, well, that is simply how mail works: put the message in a file where the next program in the pipeline will find it and pass it on. In fact, Mailman is capable of filling pretty much any pipe you're likely to have available. (People very frequently complain that Mailman sends too much mail too fast.) If there's a performance problem, almost surely either your ISP is throttling you, or your MTA is misconfigured. Again, we need to know more about how Mailman is installed, what you're trying to do, and what the other software involved is (at least an MTA such as Postfix, Exim, Sendmail, or Qmail, and preferably virus and spam filters).
Steve
On 02/28/2016 03:47 AM, Stephen J. Turnbull wrote:
If there's a performance problem, almost surely either your ISP is throttling you, or your MTA is misconfigured. Again, we need to know more about how Mailman is installed, what you're trying to do, and what the other software involved is (at least an MTA such as Postfix, Exim, Sendmail, or Qmail, and preferably virus and spam filters).
When I ran it with majordomo their was no problem. all mailing lists went out instantaneously, or as fast as postfix could process it. When I send mail out with mutt, there is no problem. When I send mail out with dovecot there is no problem. The only delay I ever have is with mailman. If your saying that mailman is putting out the files efficiently, and postfix isn't picking them up, why does it then work for every other mail source, and how do I fix this interface for mailman?
Ruben
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On Sun, Feb 28, 2016 at 3:57 AM, Ruben Safir mrbrklyn@panix.com wrote:
When I ran it with majordomo their was no problem.
How was majordomo submitting email to postfix, and how is mailman now submitting email to postix?
-Jim P.
On 02/28/2016 03:47 AM, Stephen J. Turnbull wrote:
How do I speed up mailman so that it works instantaneously and stops storing messages.
If you mean "Mailman is administratively holding messages until released by a moderator," you need to tell us more about your current configuration and needs. There are several reasons why Mailman will hold messages -- what is the reason that Mailman gives for holding your messages?
No, I mean I send messages and they don't show up for a half hour even in the postfix logs....
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 07:29 AM, Ruben Safir wrote:
No, I mean I send messages and they don't show up for a half hour even in the postfix logs....
Assuming Mailman 2.1.x, the usual cause of this is a backlogged 'out' queue in Mailman. Look at Mailman's 'smtp' log, and look at how many files are in Mailman's qfiles/out/ directory (/var/spool/mailman/out in some packages).
If there are more than one or two files in the out/ queue, and if the smtp log has successive entries of the form
Feb 28 08:55:43 2016 (30307) <mesage-id> smtp to list for nnn recips, completed in t.ttt seconds
with each messages time stamp being t.ttt seconds later than the preceding message, the out/ queue is backlogged. If that is the case, Mailman isn't able to deliver fast enough to Postfix to keep up with it's volume.
Even with full VERP, Mailman should be able to deliver tens of messages per second to Postfix. If it is slower than that, there is a n issue with delivery. Once we have more info on whether the queue is backlogged and what the delivery rate is, we can say more. You might find some of the hits from the google search
site:mail.python.org inurl:mailman backlogged out queue
of interest.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
Even with full VERP, Mailman should be able to deliver tens of messages per second to Postfix. If it is slower than that, there is a n issue with delivery. Once we have more info on whether the queue is backlogged and what the delivery rate is, we can say more. You might find some of the hits from the google search
site:mail.python.org inurl:mailman backlogged out queue
of interest.
If that doesn't make it obvious what you need to do, you might want to tell us something about your configuration and use case. What version of Mailman? Did you install as a package from a distribution or from source? If from a distribution, which one?
How big are the lists (number of subscribed addresses, maximum size if there are several, and sizes can be order-of-magnitude like tens, thousands, hundred million)? How many such lists? (As Mark says, you still should be able to move up to 100,000 messages/hour with modest hardware under normal conditions.)
Are your lists configured for personalization (eg, per-user information such as subscription management URL in the footer)?
If Mailman >= 2.1.16, are your lists configured to deal with DMARC issues (bounces of posts from AOL and Yahoo! addresses)? Do you get a lot of posts from those addresses? (This can matter because DNS transactions for DMARC can take a long time if for some reason your path to those nameservers is slow.)
On 2/28/2016 1:19 PM, Stephen J. Turnbull stephen@xemacs.org wrote:
If that doesn't make it obvious what you need to do, you might want to tell us something about your configuration and use case. What version of Mailman? Did you install as a package from a distribution or from source? If from a distribution, which one?
Been meaning to ask...
Would it be difficult to add a command similar to postfix's postconf -n that would dump the currently used config?
Sure would help with troubleshooting...
On 02/28/2016 03:57 PM, Tanstaafl wrote:
On 2/28/2016 1:19 PM, Stephen J. Turnbull stephen@xemacs.org wrote:
If that doesn't make it obvious what you need to do, you might want to tell us something about your configuration and use case. What version of Mailman? Did you install as a package from a distribution or from source? If from a distribution, which one?
Been meaning to ask...
Would it be difficult to add a command similar to postfix's postconf -n that would dump the currently used config?
When I read this, it just seems to me that you guys don't know how the software works. I posted logs to postfix and what they are asking is frankly impossible to acquire. This is a live system running a lot of email in /var/spool/mail and the delays show up when the system is under use. I can't just restart it and expect the delays to show up and I can't get answers to basic questions such as
When the email comes to mailman, where does it go. How does the MTA know to pick up the mail. It seems to process mail in sweeps, rather than in real time when mail arrives.
2016-02-28T05:10:27.327566-05:00 www postfix/anvil[17919]: statistics: max connection rate 1/60s for (smtp:74.63.252.212)
+at Feb 28 05:04:45
2016-02-28T05:10:27.331943-05:00 www postfix/anvil[17919]: statistics: max connection count 1 for (smtp:74.63.252.212) at
+Feb 28 05:04:45
2016-02-28T05:10:27.336700-05:00 www postfix/anvil[17919]: statistics: max cache size 3 at Feb 28 05:08:54
2016-02-28T10:27:00.724625-05:00 www postfix/smtpd[21374]: NOQUEUE: reject: RCPT from www.mrbrklyn.com[96.57.23.82]: 450
+4.1.2 dyfet@gnutelephony.org: Recipient address rejected: Domain not found; from=hangout-bounces@nylxs.com
+to=dyfet@gnutelephony.org proto=ESMTP helo=
which logs and what time stamps?
2016-02-28T09:04:03.801283-05:00 www postfix/smtpd[20212]: NOQUEUE: reject: RCPT from unknown[162.144.221.253]: 554 5.7.1
+Service unavailable; Client host [162.144.221.253] blocked using zen.spamhaus.org;
+https://www.spamhaus.org/sbl/query/SBLCSS; from=KingstonHicks@bsbw.launchyourfirstproduct.com to=hangout@nylxs.com
+proto=ESMTP helo=
+450 4.1.2 dyfet@gnutelephony.org: Recipient address rejected: Domain not found; from=hangout-bounces@nylxs.com
+to=dyfet@gnutelephony.org proto=ESMTP helo=
That is seperate greps of TO and FROM'S to the maillist.
Obviously it is very difficult to know what is happening by looking at the logs.
Majordomo sends email straight to the MTA though aliases and a pipe.
Mailman has a dameon running. It is obviously very different. I think
rejected email is choking whatever process is delivering mail..but that is a guess.
Reuvain
Sure would help with troubleshooting...
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/mrbrklyn%40panix.com
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
www:~ # /usr/lib/mailman/bin/version Using Mailman version: 2.1.17
www:~ # postconf -d | grep mail_version mail_version = 2.11.3 milter_macro_v = $mail_name $mail_version
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Ruben Safir writes:
When I read this, it just seems to me that you guys don't know how the software works.
This is user-supported software. Not everybody is all that expert, but people contribute when and what they can. And mild thread- hijacking like asking if there's a way to get certain information related to the thread isn't unusual.
I posted logs to postfix and what they are asking is frankly impossible to acquire.
Then perhaps you have to live with the delays. They (like us) ask for information that is known to have been useful in diagnosing problems in the past. Without information, it's impossible to diagnose.
By the way, it is not unheard of that restarting the mail system clears up backlogs just like that. (Don't bet your house on it, but it does happen.) A typical reason is that in normal operation, "exponential backoff" is used so that delays from retries increase with each retry until you've tried for several days, but on system startup there's provision for an attempt to "flush" the queue right away, not worrying about when the next try for a message is scheduled. If that succeeds, the backlog disappears.
Of course if there's a persistent problem, the backlog will reappear.
I can't get answers to basic questions such as
Short answer: you didn't ask. In the posts I've seen, you just announced that it wasn't working as you expected.
When the email comes to mailman, where does it go.
First it is delivered to the mailman program itself over a pipe, according to the alias in the Postfix configuration. If it's a post, it is then saved in a .pck file in a queue directory (typically /usr/var/mailman/queue/incoming/). The incoming runner checks the directory for changes, picks up the new .pck when it appears, decides what to do with it after checking for spam, inserting footer and other mailing list stuff like List-* headers, and finally puts in in one or more queues (eg, archive, outgoing). Those runners do the same dance. Typically the whole process occurs in under a second.
How does the MTA know to pick up the mail.
The Mailman outgoing runner connects to the well-known port for mail, where the MTA is listening. I think it's possible to configure Mailman to use the "sendmail" program via stdin, but in modern systems (specifically Postfix) it's much more efficient to use a socket, since the sendmail program itself often just drops the message in a queue to be processed by a daemon serving the outgoing queue.
It seems to process mail in sweeps, rather than in real time when mail arrives.
What do you mean by "real time"? If you look at what Postfix does, it's just like Mailman: composed of several programs, each with a specific responsibility, that receives a message as a file in a queue directory, processes it, places the output in a file in another queue directory, and only if that was successful, removes the input queue file. The final step is to connect to another mail server over the network, but again the input queue file is not removed until the remote server says "250 OK", which means that the message has been saved as a file on the remote system. This can be done quickly (my system typically processes a post end-to-end in under a second, though at most 20 subscribers), but often not in what communication engineers mean by "real time".
In fact, when you see stuff like this:
2016-02-28T10:27:00.724625-05:00 www postfix/smtpd[21374]: NOQUEUE: reject: RCPT from www.mrbrklyn.com[96.57.23.82]: 450 +4.1.2 dyfet@gnutelephony.org: Recipient address rejected: Domain not found; from=hangout-bounces@nylxs.com +to=dyfet@gnutelephony.org proto=ESMTP helo=
what you're seeing is that you have quite a few possibly invalid addresses that you're trying to send to. In fact, the first ten were rejected without a single success -- I don't see how any mailing list manager could deal efficiently with such a high rate of failure, especially if it frequently involves DNS failures as this one does, and several of the other log entries report the same outcome. Note that the failure is considered temporary. IIRC, that means that DNS lookup failed with no result, not that the relevant nameserver (the .org rootserver in this case) said there was no such domain. That probably means multiple timeouts on DNS lookups, each of which might take 30 seconds. (I tried "host gnutelephony.org" myself, and got a DNS timeout after 30 seconds.) If you have two nameservers configured, it seems likely that we have just found one common reason it takes your Mailman 60+ seconds to process one queue entry.
Temporary failure also means that the queue file will remain, as the system believes that retrying may succeed. (Any mailing list manager that does not preserve the message in this way is losing mail.) I don't know precisely what you mean by "sweeps", but the fact that there are quite a few temporary failure queuefiles hanging around would account for apparently unrelated posts being processed at the same time.
That is seperate greps of TO and FROM'S to the maillist. Obviously it is very difficult to know what is happening by looking at the logs. Majordomo sends email straight to the MTA though aliases and a pipe.
I certainly hope it wouldn't do that in the cases reported in the log in your post, since the overwhelming majority are temporary failures. It needs to be prepared to save the message to retry later. I suppose it's possible that Majordomo would have dealt with whatever the situation actually is more efficiently, but I suspect that it would have problems of some kind, and you just happened to change to Mailman at a time when the list's environment became unstable.
I suspect that your problem involves your DNS, since so many of the posts are rejected with "Domain not found". Or it could be that you just have an overwhelming majority of addresses with invalid domains, or even that Postfix is misconfigured to report temporary failure (450) when it should be reporting permanent failure (nonexistent domain, 550).
By the way, when did you switch to Mailman? Did you start experiencing this problem immediately when you did?
On 02/28/2016 10:04 PM, Stephen J. Turnbull wrote:
I don't see how any mailing list manager could deal efficiently with such a high rate of failure,
OK - I understand. Another mailing list manager was doing it before perl4 really died. I guess the reality of this is beginning to sink in.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 10:04 PM, Stephen J. Turnbull wrote:
By the way, when did you switch to Mailman? Did you start experiencing this problem immediately when you did?
about 6 months ago and the problem was concurrent absolutely with the initiation of mailman.
There seems to be no settings I can set to make mailman work, if what your saying is true.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 10:31 PM, Ruben Safir wrote:
On 02/28/2016 10:04 PM, Stephen J. Turnbull wrote:
By the way, when did you switch to Mailman? Did you start experiencing this problem immediately when you did?
about 6 months ago and the problem was concurrent absolutely with the initiation of mailman.
There seems to be no settings I can set to make mailman work, if what your saying is true.
I think we can fix your issue fairly simply.
Please, as I asked in my reply at https://mail.python.org/pipermail/mailman-users/2016-February/080524.html, post the output from 'postconf -n' and the contents of mm_cfg.py.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 02/29/2016 01:34 AM, Mark Sapiro wrote:
I think we can fix your issue fairly simply.
Please, as I asked in my reply at https://mail.python.org/pipermail/mailman-users/2016-February/080524.html, post the output from 'postconf -n' and the contents of mm_cfg.py.
Sorry, I got mixed up. Its just probably the frustration. Everyone uses mailman, I don't know why I'm so stupid
alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases biff = no canonical_maps = hash:/etc/postfix/canonical command_directory = /usr/sbin config_directory = /etc/postfix content_filter = daemon_directory = /usr/lib/postfix data_directory = /var/lib/postfix debug_peer_level = 2 debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd $daemon_directory/$process_name $process_id & sleep 5 defer_transports = delay_warning_time = 1h disable_dns_lookups = no disable_mime_output_conversion = no disable_vrfy_command = yes html_directory = /usr/share/doc/packages/postfix-doc/html inet_interfaces = all inet_protocols = ipv4 mail_owner = postfix mail_spool_directory = /var/mail mailbox_command = /usr/bin/procmail mailbox_size_limit = 0 mailbox_transport = mailq_path = /usr/bin/mailq manpage_directory = /usr/share/man masquerade_classes = envelope_sender, header_sender, header_recipient masquerade_domains = mrbrklyn.com, mrbrklyn.com masquerade_exceptions = root message_size_limit = 0 message_strip_characters = \0 mydestination = www.mrbrklyn.com, www2.mrbrklyn.com, home.mrbrklyn.com, mrbrklyn.com, nylxs.com, brooklyn-living.com, freedon_it.com mydomain = mrbrklyn.com myhostname = mrbrklyn.com mynetworks_style = subnet newaliases_path = /usr/bin/newaliases queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/packages/postfix-doc/README_FILES relay_clientcerts = relayhost = relocated_maps = hash:/etc/postfix/relocated sample_directory = /usr/share/doc/packages/postfix-doc/samples sender_canonical_maps = hash:/etc/postfix/sender_canonical sendmail_path = /usr/sbin/sendmail setgid_group = maildrop smtp_enforce_tls = no smtp_generic_maps = hash:/etc/postfix/generic smtp_sasl_auth_enable = no smtp_sasl_password_maps = smtp_sasl_security_options = smtp_tls_CAfile = smtp_tls_CApath = smtp_tls_cert_file = smtp_tls_key_file = smtp_tls_session_cache_database = smtp_use_tls = no smtpd_banner = $myhostname ESMTP smtpd_client_restrictions = reject_rbl_client dnsbl.sorbs.net smtpd_data_restrictions = reject_unauth_pipelining, permit smtpd_delay_reject = yes smtpd_helo_required = yes smtpd_helo_restrictions = permit_mynetworks, reject_invalid_hostname, regexp:/etc/postfix/helo.regexp, permit smtpd_recipient_restrictions = check_client_access hash:/etc/postfix/helo_client_exceptions check_sender_access hash:/etc/postfix/sender_checks, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit_mynetworks, reject_unauth_destination, permit_mynetworks, reject_unauth_destination, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net reject_rbl_client cbl.abuseat.org, permit smtpd_sasl_auth_enable = no smtpd_sender_restrictions = hash:/etc/postfix/access, reject_unknown_sender_domain smtpd_tls_CAfile = smtpd_tls_CApath = smtpd_tls_ask_ccert = no smtpd_tls_cert_file = smtpd_tls_key_file = smtpd_tls_received_header = no smtpd_use_tls = no strict_8bitmime = no strict_rfc821_envelopes = yes transport_maps = hash:/etc/postfix/transport unknown_local_recipient_reject_code = 550 virtual_alias_domains = hash:/etc/postfix/virtual virtual_alias_maps = hash:/etc/postfix/virtual
vim /usr/lib/mailman/Mailman/mm_cfg.py
###############################################
# Here's where we get the distributed defaults.
from Defaults import *
##################################################
# Put YOUR site-specific settings below this line.
DEFAULT_URL_PATTERN = 'http://%s/mailman/'
DEFAULT_NNTP_HOST = 'www.mrbrklyn.com'
DEFAULT_EMAIL_HOST = 'nylxs.com'
DEFAULT_URL_HOST = 'www.nylxs.com'
MTA = 'Postfix'
POSTFIX_ALIAS_CMD = '/usr/sbin/postalias'
POSTFIX_MAP_CMD = '/usr/sbin/postmap'
DELIVERY_MODULE = 'SMTPDirect'
SMTPHOST = 'mrbrklyn.com'
SMTPPORT = '25'
add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST)
add_virtualhost('lists.mrbrklyn.com', 'mrbrklyn.com')
IMAGE_LOGOS = '/mailmanicons/'
There is another one in apache:
I don't know if it is being used.
vim /usr/local/apache/conf/mailman/Mailman/mm_cfg.py
###############################################
# Here's where we get the distributed defaults.
from Defaults import *
##################################################
# Put YOUR site-specific settings below this line.
DEFAULT_URL_PATTERN = 'http://%s/mailman/'
DEFAULT_NNTP_HOST = 'www.mrbrklyn.com'
DEFAULT_EMAIL_HOST = 'nylxs.com'
DEFAULT_URL_HOST = 'www.nylxs.com'
MTA = 'Postfix'
POSTFIX_ALIAS_CMD = '/usr/sbin/postalias'
POSTFIX_MAP_CMD = '/usr/sbin/postmap'
DELIVERY_MODULE = 'SMTPDirect'
SMTPHOST = 'mrbrklyn.com'
SMTPPORT = '25'
add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST)
IMAGE_LOGOS = '/mailmanicons/'
--
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 10:51 PM, Ruben Safir wrote:
On 02/29/2016 01:34 AM, Mark Sapiro wrote:
I think we can fix your issue fairly simply.
Please, as I asked in my reply at https://mail.python.org/pipermail/mailman-users/2016-February/080524.html, post the output from 'postconf -n' and the contents of mm_cfg.py.
Sorry, I got mixed up. Its just probably the frustration. Everyone uses mailman, I don't know why I'm so stupid
smtpd_recipient_restrictions = check_client_access hash:/etc/postfix/helo_client_exceptions check_sender_access hash:/etc/postfix/sender_checks, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit_mynetworks, reject_unauth_destination, permit_mynetworks, reject_unauth_destination, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net reject_rbl_client cbl.abuseat.org, permit
This is almost certainly your problem. All those checks take time, especially if DNS is slow. If you send a message from a client and Postfix takes 5 seconds to accept it, it's no big deal. If Mailman sends to 10 or 20 recipients, and it takes Postfix a minute to respond, it still may be no big deal unless another two posts arrive in that minute , and so on until you have a big backlog.
I suggest that if you really want all those checks, that you set up a separate port for Mailman to send to without all those rbl lookups and recipient domain lookups. See below.
vim /usr/lib/mailman/Mailman/mm_cfg.py
############################################### # Here's where we get the distributed defaults.
from Defaults import *
################################################## # Put YOUR site-specific settings below this line. DEFAULT_URL_PATTERN = 'http://%s/mailman/' DEFAULT_NNTP_HOST = 'www.mrbrklyn.com' DEFAULT_EMAIL_HOST = 'nylxs.com' DEFAULT_URL_HOST = 'www.nylxs.com' MTA = 'Postfix' POSTFIX_ALIAS_CMD = '/usr/sbin/postalias' POSTFIX_MAP_CMD = '/usr/sbin/postmap' DELIVERY_MODULE = 'SMTPDirect' SMTPHOST = 'mrbrklyn.com' SMTPPORT = '25'
Here's where I'm suggesting changes. Pick a port, say 8000, although it could be anything that doesn't conflict.
Then change the above to
SMTPHOST = '127.0.0.1' SMTPPORT = 8000
(don't quote the port - it's a number, not a string)
Also, while you're at it I suggest adding
VERP_PASSWORD_REMINDERS = Yes VERP_PERSONALIZED_DELIVERIES = Yes VERP_DELIVERY_INTERVAL = 1
for more reliable bounce processing.
But, see below for changes to Postfix master.cf that you must make first.
add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('lists.mrbrklyn.com', 'mrbrklyn.com') IMAGE_LOGOS = '/mailmanicons/'
There is another one in apache: I don't know if it is being used. vim /usr/local/apache/conf/mailman/Mailman/mm_cfg.py
No, that shouldn't be used.
In Postfix master.cf add the following stanza
127.0.0.1:8000 inet n - - - - smtpd -o smtpd_authorized_xforward_hosts=127.0.0.0/8 -o mynetworks=127.0.0.0/8 -o smtpd_recipient_restrictions=permit_mynetworks,reject -o smtpd_client_restrictions= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_data_restrictions=
Make this addition to Postfix master.cf and reload Postfix. Only after you've done that and Postfix is listening on the loopback interface port 8000, make the changes to mm_cfg.py and restart Mailman.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 02/28/2016 11:19 PM, Mark Sapiro wrote:
Also, while you're at it I suggest adding
VERP_PASSWORD_REMINDERS = Yes VERP_PERSONALIZED_DELIVERIES = Yes VERP_DELIVERY_INTERVAL = 1
for more reliable bounce processing.
One more thing. If you add the above to mm_cfg.py, you also need
recipient_delimiter = +
in Postfix main.cf. If that's a problem, then don't add those 3 lines to mm_cfg.py.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 02/29/2016 02:19 AM, Mark Sapiro wrote:
On 02/28/2016 10:51 PM, Ruben Safir wrote:
On 02/29/2016 01:34 AM, Mark Sapiro wrote:
I think we can fix your issue fairly simply.
Please, as I asked in my reply at https://mail.python.org/pipermail/mailman-users/2016-February/080524.html, post the output from 'postconf -n' and the contents of mm_cfg.py.
Sorry, I got mixed up. Its just probably the frustration. Everyone uses mailman, I don't know why I'm so stupid
smtpd_recipient_restrictions = check_client_access hash:/etc/postfix/helo_client_exceptions check_sender_access hash:/etc/postfix/sender_checks, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit_mynetworks, reject_unauth_destination, permit_mynetworks, reject_unauth_destination, reject_invalid_hostname, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net reject_rbl_client cbl.abuseat.org, permit
This is almost certainly your problem. All those checks take time, especially if DNS is slow. If you send a message from a client and Postfix takes 5 seconds to accept it, it's no big deal. If Mailman sends to 10 or 20 recipients, and it takes Postfix a minute to respond, it still may be no big deal unless another two posts arrive in that minute , and so on until you have a big backlog.
I suggest that if you really want all those checks, that you set up a separate port for Mailman to send to without all those rbl lookups and recipient domain lookups. See below.
vim /usr/lib/mailman/Mailman/mm_cfg.py
############################################### # Here's where we get the distributed defaults.
from Defaults import *
################################################## # Put YOUR site-specific settings below this line. DEFAULT_URL_PATTERN = 'http://%s/mailman/' DEFAULT_NNTP_HOST = 'www.mrbrklyn.com' DEFAULT_EMAIL_HOST = 'nylxs.com' DEFAULT_URL_HOST = 'www.nylxs.com' MTA = 'Postfix' POSTFIX_ALIAS_CMD = '/usr/sbin/postalias' POSTFIX_MAP_CMD = '/usr/sbin/postmap' DELIVERY_MODULE = 'SMTPDirect' SMTPHOST = 'mrbrklyn.com' SMTPPORT = '25'
Here's where I'm suggesting changes. Pick a port, say 8000, although it could be anything that doesn't conflict.
Then change the above to
SMTPHOST = '127.0.0.1' SMTPPORT = 8000
(don't quote the port - it's a number, not a string)
Also, while you're at it I suggest adding
VERP_PASSWORD_REMINDERS = Yes VERP_PERSONALIZED_DELIVERIES = Yes VERP_DELIVERY_INTERVAL = 1
for more reliable bounce processing.
But, see below for changes to Postfix master.cf that you must make first.
add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('lists.mrbrklyn.com', 'mrbrklyn.com') IMAGE_LOGOS = '/mailmanicons/'
There is another one in apache: I don't know if it is being used. vim /usr/local/apache/conf/mailman/Mailman/mm_cfg.py
No, that shouldn't be used.
In Postfix master.cf add the following stanza
127.0.0.1:8000 inet n - - - - smtpd -o smtpd_authorized_xforward_hosts=127.0.0.0/8 -o mynetworks=127.0.0.0/8 -o smtpd_recipient_restrictions=permit_mynetworks,reject -o smtpd_client_restrictions= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_data_restrictions=
Make this addition to Postfix master.cf and reload Postfix. Only after you've done that and Postfix is listening on the loopback interface port 8000, make the changes to mm_cfg.py and restart Mailman.
OK . That port is restricted to a 12.0.0.0/8 relay? The last thing I need is for someone to be monitoring this list and pounding port 8000 for a spam relay.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Mark Sapiro writes:
This is almost certainly your problem. All those [RBL] checks take time, especially if DNS is slow.
Also, several of the checks (RBL and otherwise) seem to be in there repeatedly. ISTR that Postfix does the checks in the order specified, so they might be done multiple times. That doesn't seem useful. :-(
Ruben Safir writes:
On 02/29/2016 01:42 AM, Ruben Safir wrote:
dig gnutelephony.org
that does hang. These are OLD mailing lists and it is hard to be responsible for 20 years of DNS errors by organizations I have no control over.
You don't have to. Mailman will do that for you if you have bounce processing turned on and configured appropriately. But looking at the rejected bogus addresses[1] in your Postfix log suggests to me that you may have been targeted by spammers who are trying to harvest addresses from your mailflow. It's quite possible that those were added very recently (within days) by botnets, or that your bounce processing isn't configured appropriately for the traffic you're getting. If the latter, the bogus addresses could have been there for a long time.
Like Carl, I recommend cleaning up your lists. To get one cleaned quickly, if you don't have the time to do it by hand, Mailman can do a pretty good job automatically. Go to the administration interface for each list, select bounce processing, and set
bounce_processing = Yes # Yes
bounce_score_threshold = 0 # 5.0
bounce_you_are_disabled_warnings = 1 # 3
bounce_you_are_disabled_warnings_interval = 1 # 7
bounce_notify_owner_on_disable = No # Yes
bounce_notify_owner_on_removal = No # Yes
Values at the end of the line are defaults.and wait a day or two (I'm assuming you have traffic on each list daily, if not, send a test message each day). After the first day, all the bogus addresses (and most likely a few valid ones) will be disabled, and no mail will be sent to them, so Postfix will do no DNS lookups for them.
After the second day, they'll be gone and you'll have clean lists. You may also have some irate ex-subscribers[2]; if you care about that, adjust the number of disabled warnings and the disabled warnings interval upward to give them a better chance to respond that they're real. 2 and 2 should do, but you'll have to wait a week for the lists to be actually cleaned. And some will ignore the warnings or not receive them and get unsubscribed -- if you care about that, you have to do it by hand.
After that, you should go through and reenable any real subscribers who got disabled. And you probably want to set the values back to something closer to the defaults. The defaults should be enough to clean out any invalid addresses that "just happen" to get added within a couple weeks. If having done this, you get infested with several invalid addresses quickly, you probably should tighten up the subscription policy. :-(
If you have a lot of lists, it's possible to write scripts to do this kind of configuration automatically for a list of lists, but I haven't done it so can't help out beyond saying it can be done.
Footnotes: [1] Yes, gnutelephony.org looks like it's probably something that was alive and died because it got no love, but check out the others to see what I mean.
[2] Mail really is quite unreliable. Occasionally you do run into an "undeliverable" result for a perfectly good address. If that happens to a subscriber, they'll get a bounce, and with the aggressive settings I propose, they get disabled immediately, and unsubscribed if it happens again the next day.
Mark Sapiro writes:
This is almost certainly your problem. All those [RBL] checks take time, especially if DNS is slow.
Also, several of the checks (RBL and otherwise) seem to be in there repeatedly. ISTR that Postfix does the checks in the order specified, so they might be done multiple times. That doesn't seem useful. :-(
Ruben Safir writes:
On 02/29/2016 01:42 AM, Ruben Safir wrote:
dig gnutelephony.org
that does hang. These are OLD mailing lists and it is hard to be responsible for 20 years of DNS errors by organizations I have no control over.
You don't have to. Mailman will do that for you if you have bounce processing turned on and configured appropriately. But looking at the rejected bogus addresses[1] in your Postfix log suggests to me that you may have been targeted by spammers who are trying to harvest addresses from your mailflow. It's quite possible that those were added very recently (within days) by botnets, or that your bounce processing isn't configured appropriately for the traffic you're getting. If the latter, the bogus addresses could have been there for a long time.
Like Carl, I recommend cleaning up your lists. To get one cleaned quickly, if you don't have the time to do it by hand, Mailman can do a pretty good job automatically. Go to the administration interface for each list, select bounce processing, and set
bounce_processing = Yes # Yes
bounce_score_threshold = 0.1 # 5.0
bounce_you_are_disabled_warnings = 1 # 3
bounce_you_are_disabled_warnings_interval = 1 # 7
bounce_notify_owner_on_disable = No # Yes
bounce_notify_owner_on_removal = No # Yes
Values at the end of the line are defaults.and wait a day or two (I'm assuming you have traffic on each list daily, if not, send a test message each day). After the first day, all the bogus addresses (and most likely a few valid ones) will be disabled, and no mail will be sent to them, so Postfix will do no DNS lookups for them. After the second day, they'll be gone and you'll have clean lists.
You may also have some irate ex-subscribers[2]; if you care about that, adjust the number of disabled warnings and the disabled warnings interval upward to give them a better chance to respond that they're real. 2 and 2 should do, but you'll have to wait a week for the lists to be actually cleaned. And some will ignore the warnings or not receive them and get unsubscribed -- if you care about that, you have to do it by hand.
After that, you should go through and reenable any real subscribers who got disabled. And you probably want to set the values back to something closer to the defaults. The defaults should be enough to clean out any invalid addresses that "just happen" to get added within a couple weeks. If having done this, you get infested with several invalid addresses quickly, you probably should tighten up the subscription policy. :-(
If you have a lot of lists, it's possible to write scripts to do this kind of configuration automatically for a list of lists, but I haven't done it so can't help out beyond saying it can be done.
Footnotes: [1] Yes, gnutelephony.org looks like it's probably something killed because it got no love, but check out the others to see what I mean.
[2] Mail really is quite unreliable. Occasionally you do run into an "undeliverable" result for a perfectly good address. If that happens to a subscriber, they'll get a bounce, and with the aggressive settings I propose, they get disabled immediately, and unsubscribed if it happens again the next day.
On 2/28/2016 11:19 PM, Mark Sapiro wrote:
On 02/28/2016 10:51 PM, Ruben Safir wrote:
smtpd_recipient_restrictions = check_client_access hash:/etc/postfix/helo_client_exceptions check_sender_access hash:/etc/postfix/sender_checks, reject_invalid_hostname, [...]
This is almost certainly your problem. All those checks take time, especially if DNS is slow.
All this suggests an overly-complex email setup. If the goal is spam rejection, consider using spam-assassin or even pipe the email through on of the paid on-line services; don't use the postfix address filters for this. You'll find all manner of docs online describing how to set this up.
As in any discovery effort- it it was "working well" and isn't any more, what changed?
Is your internet connection classed as residential or business? The former can have all manner of semi-secret filter/throttling applied.
Is your DNS forwarding to the cable company's server(s) or another, such as google (8.8.8.8)? When I had cable internet, the local servers were terrible and -their- support eventually suggested I use that company's DNS in another region and gave me those addresses.
Oh, did you do any of the simple tests like look at the timing when you manually send to some of the failing addresses? (I don't recall seeing this mentioned.)
Later,
z!
On 02/28/2016 10:04 PM, Stephen J. Turnbull wrote:
IIRC, that means that DNS lookup failed with no result, not that the relevant nameserver (the .org rootserver in this case) said there was no such domain. That probably means multiple timeouts on DNS lookups, each of which might take 30 seconds. (I tried "host gnutelephony.org" myself, and got a DNS timeout after 30 seconds.)
It is a strange thought but failure of one message shouldn't prevent the new messages in mailman to go out in real time. The definition of real time is immediately when the mail is received, it is processed and sent through. A sweep, which is how it currently behaves, means that mailman holds on to all its messages, maybe for 10 minutes, and then shoots them all through to postfix at that point, and then it stops doing sending again until it feels like it. What I want to set iut up to do, if it can, is send the messages out right away as soon as it gets them. I want it to ignore the time outs and bounces and just send the messages out right away, when they arrive and after post processing the headers etc.
I don't care if it takes 30 seconds to do dig gnutelephony.org. It should just ignore that sent message and not do anything if it times out. I'm running mailing lists on coding and meetings and politics, not running 911.
Can it be set up like that and if so, please tell me how.
Ruben
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 10:42 PM, Ruben Safir wrote:
Can it be set up like that and if so, please tell me how.
I'm trying to!
Please see my posts archived at https://mail.python.org/pipermail/mailman-users/2016-February/080524.html and https://mail.python.org/pipermail/mailman-users/2016-February/080533.html.
Please as I ask, show me 'postconf -n' mm_cfg.py and you might as well include Postfix's master.cf too.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 02/29/2016 01:50 AM, Mark Sapiro wrote:
On 02/28/2016 10:42 PM, Ruben Safir wrote:
Can it be set up like that and if so, please tell me how.
I'm trying to!
Please see my posts archived at https://mail.python.org/pipermail/mailman-users/2016-February/080524.html and https://mail.python.org/pipermail/mailman-users/2016-February/080533.html.
Please as I ask, show me 'postconf -n' mm_cfg.py and you might as well include Postfix's master.cf too.
I did ... sorry. I'm trying to not let my frustration show. I apologize if my tone can be better.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/29/2016 01:42 AM, Ruben Safir wrote:
dig gnutelephony.org
that does hang. These are OLD mailing lists and it is hard to be responsible for 20 years of DNS errors by organizations I have no control over.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 2/28/2016 10:53 PM, Ruben Safir wrote:
On 02/29/2016 01:42 AM, Ruben Safir wrote:
dig gnutelephony.org
that does hang. These are OLD mailing lists and it is hard to be responsible for 20 years of DNS errors by organizations I have no control over.
I think that may be the crux of your problem. If half a list of 500 is addresses that timeout instead of being immediately rejected, well, 30 seconds each times 250 timeouts equals 2 hours wasted.
I would run through the logs to find all the domains that are timing out on dns, then mark the recipients on those domains as "no mail". After that, see how delivery goes for the rest of them. Besides, why keep users on the list than don't exist or actually get delivery?
Oh, and -nothing- involved in email handling is "real time", which has a fairly specific meaning in computing. *All* email is queued at least a couple of times along the way and delivered as those systems get around to it. Often that's within seconds, but not always. I'm on some lists with 100s of users, and it's not uncommon for a message to take an hour to get to everyone.
z!
On 02/29/2016 02:12 AM, Carl Zwanzig wrote:
Oh, and -nothing- involved in email handling is "real time", which has a fairly specific meaning in computing. *All* email is queued at least a couple of times along the way and delivered as those systems get around to it.
Use whatever terminology you wish. RT is a scheduler term, but it has meaning outside of that. I can't have mailing list mail waiting for an hour to reach users. It kills the flow of discussion.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
[changed the subject, I don't intend to carry this topic any further]
On 2/28/2016 11:28 PM, Ruben Safir wrote:
On 02/29/2016 02:12 AM, Carl Zwanzig wrote:
Oh, and -nothing- involved in email handling is "real time", which has a fairly specific meaning in computing.
Use whatever terminology you wish. RT is a scheduler term, but it has meaning outside of that. I can't have mailing list mail waiting for an hour to reach users. It kills the flow of discussion.
Email is not, and never has been, "real time" in any use of the term as applied to computing. Granted, we're not talking about calculating machine control parameters in precise 20ms windows, but as multiple people have said, email is always going to store and forward which is not RT- it might take seconds or it might take hours. This doesn't kill the discussion on a great many lists out there, some with 1000s of users. (Some think this is a feature, not a bug, in that writers are likely to spend more time crafting their response. This particular message has taken about 30 minutes elapsed time to write. Anyway, it's you list, not mine.)
https://en.wikipedia.org/wiki/Real-time_computing "In computer science, real-time computing (RTC), or reactive computing describes hardware and software systems subject to a "real-time constraint", for example from event to system response."
'The term "near real-time" or "nearly real-time" (NRT), in telecommunications and computing, refers to the time delay introduced, by automated data processing or network transmission, between the occurrence of an event and the use of the processed data, such as for display or feedback and control purposes. For example, a near-real-time display depicts an event or situation as it existed at the current time minus the processing time, as nearly the time of the live event."
Another definition says "the actual time during which a process takes place or an event occurs." And if that process takes an hour with no added delays, that's still "in real time".
If you really need practically immediate communication, then one of the various chat systems might be a better solution. If you want to use email and not hassle about the configs, you might also consider one other hosted (and paid) mailman services. See the FAQ on that.
z!
<snip>
I would run through the logs to find all the domains that are timing out on dns, then mark the recipients on those domains as "no mail". After that, see how delivery goes for the rest of them. Besides, why keep users on the list than don't exist or actually get delivery?
Shouldn't mailman (or any good mailing list server) be automatically removing bouncing addresses? I can see that if the old mailing lists weren't that advanced that you could have 20 years of buildup of bad addresses, but shouldn't mailman have unsubscribed these addresses after 6 months of bouncing? I know on the several mailing lists I administer that bad addresses get auto-unsubscribed.
And, if there are that many bad/bouncing addresses on the list in question, how many honeypots are out there flagging this sender as a bad sender and harming the listserv, or possibly the whole domain or IP as being a problem sender?
Oh, and -nothing- involved in email handling is "real time", which has a fairly specific meaning in computing. *All* email is queued at least a couple of times along the way and delivered as those systems get around to it. Often that's within seconds, but not always. I'm on some lists with 100s of users, and it's not uncommon for a message to take an hour to get to everyone.
Definitely. It is important for people who are using email at all professionally to understand that email delivery is *NOT* instantaneous despite appearances. And that email delivery regularly takes a few minutes to an hour, but can, under absolutely normal circumstances, take 3 or 4 days. And that the US Government has legislation that defines normal email delivery to be within 30 days (yes, a whole month)...
--
from my mac to yours...
Keith Seyffarth mailto:weif@weif.net http://www.weif.net/ - Home of the First Tank Guide! http://www.rpgcalendar.net/ - the Montana Role-Playing Calendar
http://www.miscon.org/ - Montana's Longest Running Science Fiction Convention
On 02/29/2016 10:44 AM, Keith Seyffarth wrote:
how many honeypots are out there flagging this sender as a bad sender and harming the listserv, or possibly the whole domain or IP as being a problem sender?
there are no honey pots. Most of the email addresses have been on the lists for a decade or more. Almost no additions have been made, and most of the address are people I've shook hands with.
gnutelephony, for example, is David Sugar. Brilliant coder.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/29/2016 07:44 AM, Keith Seyffarth wrote:
Shouldn't mailman (or any good mailing list server) be automatically removing bouncing addresses?
Mailman is very good at this IF it is properly configured. There are list settings under control of the list admin. The defaults are reasonable for lists that moderate to high traffic volumes but probably need to be tuned for low volume lists.
Also, the site can enable VERP like envelope senders for better bounce recognition, although the standard and heuristic recognizers are pretty good.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Shouldn't mailman (or any good mailing list server) be automatically removing bouncing addresses?
Mailman is very good at this IF it is properly configured. There are list settings under control of the list admin. The defaults are reasonable for lists that moderate to high traffic volumes but probably need to be tuned for low volume lists.
I haven't had any problems with my many low-volume lists correctly removing dead addresses - as long as the receiving MTA is not configured to not bounce undeliverable mail properly...
Keith
--
from my mac to yours...
Keith Seyffarth mailto:weif@weif.net http://www.weif.net/ - Home of the First Tank Guide! http://www.rpgcalendar.net/ - the Montana Role-Playing Calendar
http://www.miscon.org/ - Montana's Longest Running Science Fiction Convention
On 02/29/2016 12:48 PM, Keith Seyffarth wrote:
I haven't had any problems with my many low-volume lists correctly removing dead addresses - as long as the receiving MTA is not configured to not bounce undeliverable mail properly...
It depends on the list and settings. The defaults are bounce_score_threshold = 5 and bounce_info_stale_after = 7. This means a user's delivery won't be disabled by bounce until the list has received bounces on 5 separate days with no more than 7 days between bounces. With these settings, a list which receives posts on average less than one day a week, will never see anyone's delivery disabled by bounce or removed by bounce processing.
For low volume lists, a lower threshold and a longer bounce_info_stale_after are often more appropriate.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The esteemed Ruben Safir has said:
On 02/28/2016 03:57 PM, Tanstaafl wrote:
On 2/28/2016 1:19 PM, Stephen J. Turnbull stephen@xemacs.org wrote:
If that doesn't make it obvious what you need to do, you might want to tell us something about your configuration and use case. What version of Mailman? Did you install as a package from a distribution or from source? If from a distribution, which one?
Been meaning to ask...
Would it be difficult to add a command similar to postfix's postconf -n that would dump the currently used config?
When I read this, it just seems to me that you guys don't know how the software works. I posted logs to postfix and what they are asking is frankly impossible to acquire. This is a live system running a lot of email in /var/spool/mail and the delays show up when the system is under use. I can't just restart it and expect the delays to show up and I can't get answers to basic questions such as
When the email comes to mailman, where does it go. How does the MTA know to pick up the mail. It seems to process mail in sweeps, rather than in real time when mail arrives.
<snip>
I'm going to chime in here, as a Mailman user, not as a list advisor. Mark and Steve know perfectly well how the software works. So do a good many others who are reading this list. I don't expect them to give you a 3-semester-hour tutorial on E-mail basics. They have asked you a few fundamental questions, which I don't see being answered. To wit:
a. What is your computer hardware?
b. What operating system are you running?
c. What is your current DNS setup? (I presume a couple of ISP servers
not on your site).
d. Where did you get your Mailman package? Prebuilt package for your
O/S? Build and install from Mailman source?
e. What are the contents of your mm_cfg.py file?
f. What's the volume of your outgoing mail? That's number of
messages/hr, average message size, number of users receiving mail.
And how much other mail is your MTA handling along with Mailman mail?
g. What is your connection to the internet backbone? And what is your
upload speed?
From looking at your Postfix logs, it looks as though you have a major DNS problem. I suggest that you install bind and set up a caching DNS server on the same box that is running Mailman. That will stop dependence on external servers, which aren't set up to handle the floods of requests that Mailman can generate. Not only is that a matter of common courtesy to outside sysadmins, any request flooding is going to be handled much more quickly on your own box.
I'm not familiar with Postfix and its logs, as I run Sendmail. However, I can tell you that reading the MTA logs carefully to see if initial attempts to connect are failing, and asking "why," if they are, can go a long way toward solving delay problems. Remember that MTA's queue connection failures for retry later, and those retries happen on 15 minute intervals in Sendmail default installations. Mailman has logging for its side of delivery to the MTA, as Mark has told you. But I'll suggest looking at MTA performance first. Keep in mind that while batching mail with a large RCPT list is efficient for some of the large target sites, those sites often simply defer back with 4.xx responses unless you cut down the number of RCPT's.
If you haven't got a clear understanding of how SMTP E-mail works, there are plenty of sources for study and learning. One thing SMTP is not, is "instantaneous," or even quick. After all, it was developed for slow-speed telephone lines a long time ago.
Hank
On 02/29/2016 03:08 AM, Hank van Cleef wrote:
a. What is your computer hardware?
fit/pc2 Intel Atom with a gig of ram and 2 gigs of swap on a tetrabyte ----- /proc/cpuinfo ----- processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 28 model name : Intel(R) Atom(TM) CPU Z510 @ 1.10GHz stepping : 2 microcode : 0x211 cpu MHz : 1100.000 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fdiv_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe nx constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 xtpr pdcm movbe lahf_lm dtherm bogomips : 2194.49 clflush size : 64 cache_alignment : 64 address sizes : 32 bits physical, 32 bits virtual power management:
although I have run the lists previously on a PentiumII
b. What operating system are you running?
opensuse 13.2 upgraded from Slackware 3.4
c. What is your current DNS setup? (I presume a couple of ISP servers not on your site).
whois nylxs.com .... Tech Email: ruben@mrbrklyn.com Name Server: WWW2.MRBRKLYN.COM Name Server: NS1.LINUXMAFIA.COM
www:~ # dig mx nylxs.com
; <<>> DiG 9.9.2 <<>> mx nylxs.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15524 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;nylxs.com. IN MX
;; ANSWER SECTION: nylxs.com. 86400 IN MX 10 www2.mrbrklyn.com.
;; AUTHORITY SECTION: nylxs.com. 86400 IN NS ns1.linuxmafia.com. nylxs.com. 86400 IN NS www2.mrbrklyn.com.
;; ADDITIONAL SECTION: www2.mrbrklyn.com. 86400 IN A 96.57.23.82 ns1.linuxmafia.com. 63442 IN A 198.144.195.186
;; Query time: 3 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Mon Feb 29 03:45:12 2016 ;; MSG SIZE rcvd: 143
d. Where did you get your Mailman package? Prebuilt package for your O/S? Build and install from Mailman source?
probably from the opensuse repos
e. What are the contents of your mm_cfg.py file?
see previous emails
f. What's the volume of your outgoing mail? That's number of messages/hr, average message size, number of users receiving mail. And how much other mail is your MTA handling along with Mailman mail?
maybe 20K emails an hour or less.
g. What is your connection to the internet backbone? And what is your upload speed?
Cable with a 50 Mbps downloads and 25 Mbps uploads connection... in theory. Faster than the DLS I used to have with Covad before 9-11.
Reuvain
So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
Thanks for the suggestions, Hank!
Hank van Cleef writes:
From looking at your Postfix logs, it looks as though you have a major DNS problem. I suggest that you install bind and set up a caching DNS server on the same box that is running Mailman.
Good idea, but I suspect that wouldn't help enough, as the test I did on gnutelephony.org timed out. You don't cache a timeout, you retry (or if you're human, you curse the spammers and remove the address). That was at home, now, with a university-grade network connection, I'm getting NXDOMAIN. It takes ~10s (middle of 3), though, which is a pretty substantial delay.[1]
Also, IIRC, most modern Linux distros already do have a cache in the libc resolver code anyway. I don't know if it's big enough to handle Mailman-sized request floods, though.
Footnotes: [1] I wonder if the root nameservers are getting hammered?
On 02/29/2016 04:19 AM, Stephen J. Turnbull wrote:
From looking at your Postfix logs, it looks as though you have a major DNS problem. I suggest that you install bind and set up a caching DNS server on the same box that is running Mailman.
Good idea, but I suspect that wouldn't help enough, as the test I did on gnutelephony.org timed out. You don't cache a timeout, you retry (or if you're human, you curse the spammers and remove the address). That was at home, now, with a university-grade network connection, I'm getting NXDOMAIN. It takes ~10s (middle of 3), though, which is a pretty substantial delay.[1]
bind is running on the server and is the primary DNS and has been for a little over a decade?
Bind and the DNS can handle the workflow, or it has since 1998. I don't believe it handles a single connection at a time...
and neither does postfix. I'll try to run everything on 12.0.0.1 8000 like was recommended, but truly it doesn't make sense to me that any of these things should cause such a slowdown. Postfix used to handle nearly 500 simultaneous outbound requests with no problem, regardless how many bad email addresses are present. failure is PART of email, and not an exceptional occurrence.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 12:57 PM, Tanstaafl wrote:
Would it be difficult to add a command similar to postfix's postconf -n that would dump the currently used config?
Mailman's
bin/config_list -o - listname | grep -v '^#'
will dump the current config. If you really want postconf -n like output that shows only settings different from the Defaults.py/mm_cfg.py settings, there's nothing currently available that I'm aware of, but I'll think about a script.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 02/28/2016 05:55 PM, Mark Sapiro wrote:
bin/config_list -o - listname | grep -v '^#'
real_name = 'hangout'
owner = ['me@here.com']
moderator = []
description = 'NYLXS Discussions List'
info = ''
subject_prefix = '[Hangout-NYLXS] '
anonymous_list = False
first_strip_reply_to = False
reply_goes_to_list = 1
reply_to_address = ''
umbrella_list = False
umbrella_member_suffix = '-owner'
send_reminders = 0
welcome_msg = """Welcome to the NYLXS Mailing List See http://www.nylxs.com Free Linux"""
send_welcome_msg = True
goodbye_msg = ''
send_goodbye_msg = True
admin_immed_notify = 0
admin_notify_mchanges = 1
respond_to_post_requests = 0
emergency = 0
-- new_member_options = 256
administrivia = True
max_message_size = 0
admin_member_chunksize = 300
host_name = 'nylxs.com'
include_rfc2369_headers = 1
include_list_post_header = 1
include_sender_header = 1
max_days_to_hold = 1
preferred_language = 'en'
available_languages = ['en']
encode_ascii_prefixes = 0
nondigestable = True
msg_header = ''
msg_footer = """_______________________________________________ %(real_name)s mailing list %(real_name)s@%(host_name)s http://www.nylxs.com/"""
scrub_nondigest = False
regular_exclude_lists = []
regular_exclude_ignore = True
regular_include_lists = []
digestable = True
digest_is_default = False
mime_is_default_digest = False
digest_size_threshhold = 30
digest_send_periodic = True
digest_header = ''
digest_footer = """_______________________________________________ %(real_name)s mailing list %(real_name)s@%(host_name)s %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s"""
digest_volume_frequency = 1
advertised = 1
subscribe_policy = 1
unsubscribe_policy = 0
ban_list = []
private_roster = 1
obscure_addresses = 1
default_member_moderation = 0
member_moderation_action = 0
member_moderation_notice = ''
accept_these_nonmembers = ['someone@world.org']
hold_these_nonmembers = []
reject_these_nonmembers = [] discard_these_nonmembers = []
generic_nonmember_action = 3
forward_auto_discards = 0
nonmember_rejection_notice = ''
require_explicit_destination = 1
acceptable_aliases = ''
max_num_recipients = 3
header_filter_rules = []
bounce_matching_headers = """ to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
bounce_processing = 1
bounce_score_threshold = 5.0
bounce_info_stale_after = 7
bounce_you_are_disabled_warnings = 2
bounce_you_are_disabled_warnings_interval = 3
bounce_unrecognized_goes_to_list_owner = 0
bounce_notify_owner_on_disable = True
bounce_notify_owner_on_removal = True
archive = 1
archive_private = 0
archive_volume_frequency = 1
nntp_host = ''
linked_newsgroup = ''
gateway_to_news = 0
gateway_to_mail = 0
news_moderation = 0
news_prefix_subject_too = 1
autorespond_postings = 0
autoresponse_postings_text = ''
autorespond_admin = 0
autoresponse_admin_text = ''
autorespond_requests = 0
autoresponse_request_text = ''
autoresponse_graceperiod = 90
filter_content = 1
filter_mime_types = ''
pass_mime_types = ''
filter_filename_extensions = """exe bat cmd com pif scr vbs cpl"""
pass_filename_extensions = ''
collapse_alternatives = 0
convert_html_to_plaintext = True
filter_action = 0
topics_enabled = 0
topics_bodylines_limit = 5
topics = []
So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 03:04 PM, Ruben Safir wrote:
On 02/28/2016 05:55 PM, Mark Sapiro wrote:
bin/config_list -o - listname | grep -v '^#'
Ruben,
This sub-thread is unrelated to your questions.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
Even with full VERP, Mailman should be able to deliver tens of messages per second to Postfix. If it is slower than that, there is a n issue with delivery. Once we have more info on whether the queue is backlogged and what the delivery rate is, we can say more. You might find some of the hits from the google search
site:mail.python.org inurl:mailman backlogged out queue
of interest.
@Ruben: If that doesn't make it obvious what you need to do, you might want to tell us something about your configuration and use case. What version of Mailman? Did you install as a package from a distribution or from the source code? If from a distribution, which one?
How big are the lists (number of subscribed addresses, maximum size if there are several, and sizes can be order-of-magnitude like tens, thousands, hundred million)? How many such lists? As Mark says, you still should be able to move up to 100,000 messages/hour with modest hardware under normal conditions. You normally have to have a huge backlog (thousands of individual messages in the outgoing queue) to see significant slowdown -- a couple of hundred should clear within a few minutes. Note: one post typically translates to several outgoing messages, up to the number of enabled subscribers for a list with full personalization.
Are your lists configured for personalization (eg, per-user information such as subscription management URL in the footer)?
On 02/28/2016 12:33 PM, Mark Sapiro wrote:
On 02/28/2016 07:29 AM, Ruben Safir wrote:
No, I mean I send messages and they don't show up for a half hour even in the postfix logs....
Assuming Mailman 2.1.x, the usual cause of this is a backlogged 'out' queue in Mailman. Look at Mailman's 'smtp' log, and look at how many files are in Mailman's qfiles/out/ directory (/var/spool/mailman/out in some packages).
If there are more than one or two files in the out/ queue, and if the smtp log has successive entries of the form
there are 38 files in there at preseent -rw-rw---- 1 mailman mailman 4441 Feb 28 16:57 1456690111.419805+dd7dabe9420b90aad8e72ee0607a4eb015fb9976.pck -rw-rw---- 1 mailman mailman 21731 Feb 28 16:33 1456695234.857981+952d2cfb1cfa9559943dc335ca8eced7d26aaa6f.pck -rw-rw---- 1 mailman mailman 30549 Feb 28 16:33 1456695235.935237+55c1c60ede5771fa2960a0460011f82266dbcb8b.pck -rw-rw---- 1 mailman mailman 26009 Feb 28 16:33 1456695236.069412+c268ffc9dfded5c93d3b3048831f058e184dd576.pck www:/var/lib/mailman/qfiles/out # ls -al |wc 38 335 4028
Feb 28 08:55:43 2016 (30307) <mesage-id> smtp to list for nnn recips, completed in t.ttt seconds
this is the postfix mail log?
like this?
2016-02-28T17:17:14.802128-05:00 www postfix/qmgr[21586]: 0F2FB1616D8: from=hangout-bounces@nylxs.com, size=6960, nrcpt=41 (queue active) 2016-02-28T17:17:14.857414-05:00 www postfix/qmgr[21586]: 7C884163D99: from=hangout-bounces@nylxs.com, size=4936, nrcpt=42 (queue active) 2016-02-28T17:17:14.876153-05:00 www postfix/qmgr[21586]: 7E61E1616C8: from=hangout-bounces@nylxs.com, size=5354, nrcpt=40 (queue active) 2016-02-28T17:17:14.893215-05:00 www postfix/qmgr[21586]: 15E131617B9: from=hangout-bounces@nylxs.com, size=3740, nrcpt=41 (queue active)
and this 2016-02-28T17:17:43.841701-05:00 www postfix/local[667]: 47291163DA1: to=hangout-bounces@nylxs.com, relay=local, delay=0.55, delays=0.03/0/0/0.51, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman bounces hangout)
18260985136, Hostname=BLUPR09MB0181.namprd09.prod.outlook.com] 12753 bytes in 0.183, 67.784 KB/sec Queued mail for delivery) 2016-02-28T15:51:56.553321-05:00 www postfix/local[25454]: E763B163D9C: to=hangout-bounces@nylxs.com, relay=local, delay=0.61, delays=0.1/0/0/0.51, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman bounces hangout) 2016-02-28T15:52:24.710239-05:00 www postfix/local[25458]: 18326163D9A: to=hangout-bounces@nylxs.com, relay=local, delay=0.61, delays=0.09/0/0/0.51, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman bounces hangout) 2016-02-28T16:06:18.504031-05:00 www postfix/local[25619]: A874B163DA1: to=hangout@nylxs.com, relay=local, delay=1.4, delays=0.88/0.02/0/0.52, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman post hangout) 2016-02-28T16:33:54.960832-05:00 www postfix/local[26033]: 0A854163DA2: to=hangout@nylxs.com, relay=local, delay=1, delays=0.17/0.05/0/0.79, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman post hangout) 2016-02-28T17:12:16.334190-05:00 www postfix/local[561]: BA2A3163DA2: to=hangout-bounces@nylxs.com, relay=local, delay=1.6, delays=0.19/0.04/0/1.4, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman bounces hangout) 2016-02-28T17:12:16.342511-05:00 www postfix/local[513]: ADAD9163DA1: to=hangout-bounces@nylxs.com, relay=local, delay=1.6, delays=0.05/0.01/0/1.6, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman bounces hangout) 2016-02-28T17:17:43.841701-05:00 www postfix/local[667]: 47291163DA1: to=hangout-bounces@nylxs.com, relay=local, delay=0.55, delays=0.03/0/0/0.51, dsn=2.0.0, status=sent (delivered to command: /usr/lib/mailman/mail/mailman bounces hangout)
with each messages time stamp being t.ttt seconds later than the preceding message, the out/ queue is backlogged. If that is the case, Mailman isn't able to deliver fast enough to Postfix to keep up with it's volume.
Even with full VERP, Mailman should be able to deliver tens of messages per second to Postfix. If it is slower than that, there is a n issue with delivery. Once we have more info on whether the queue is backlogged and what the delivery rate is, we can say more. You might find some of the hits from the google search
site:mail.python.org inurl:mailman backlogged out queue
of interest.
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 02:26 PM, Ruben Safir wrote:
On 02/28/2016 12:33 PM, Mark Sapiro wrote:
If there are more than one or two files in the out/ queue, and if the smtp log has successive entries of the form
there are 38 files in there at preseent -rw-rw---- 1 mailman mailman 4441 Feb 28 16:57 1456690111.419805+dd7dabe9420b90aad8e72ee0607a4eb015fb9976.pck -rw-rw---- 1 mailman mailman 21731 Feb 28 16:33 1456695234.857981+952d2cfb1cfa9559943dc335ca8eced7d26aaa6f.pck -rw-rw---- 1 mailman mailman 30549 Feb 28 16:33 1456695235.935237+55c1c60ede5771fa2960a0460011f82266dbcb8b.pck -rw-rw---- 1 mailman mailman 26009 Feb 28 16:33 1456695236.069412+c268ffc9dfded5c93d3b3048831f058e184dd576.pck www:/var/lib/mailman/qfiles/out # ls -al |wc 38 335 4028
This indicates a serious backlog.
Feb 28 08:55:43 2016 (30307) <mesage-id> smtp to list for nnn recips, completed in t.ttt seconds
this is the postfix mail log?
No. Look at Mailman's smtp log. This may be /var/lib/mailman/logs/smtp or /usr/local/mailman/logs/smtp or /var/log/mailman/smtp or somewhere else depending on how mailman was installed
with each messages time stamp being t.ttt seconds later than the preceding message, the out/ queue is backlogged. If that is the case, Mailman isn't able to deliver fast enough to Postfix to keep up with it's volume.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 02/28/2016 05:46 PM, Mark Sapiro wrote:
On 02/28/2016 02:26 PM, Ruben Safir wrote:
On 02/28/2016 12:33 PM, Mark Sapiro wrote:
If there are more than one or two files in the out/ queue, and if the smtp log has successive entries of the form
there are 38 files in there at preseent -rw-rw---- 1 mailman mailman 4441 Feb 28 16:57 1456690111.419805+dd7dabe9420b90aad8e72ee0607a4eb015fb9976.pck -rw-rw---- 1 mailman mailman 21731 Feb 28 16:33 1456695234.857981+952d2cfb1cfa9559943dc335ca8eced7d26aaa6f.pck -rw-rw---- 1 mailman mailman 30549 Feb 28 16:33 1456695235.935237+55c1c60ede5771fa2960a0460011f82266dbcb8b.pck -rw-rw---- 1 mailman mailman 26009 Feb 28 16:33 1456695236.069412+c268ffc9dfded5c93d3b3048831f058e184dd576.pck www:/var/lib/mailman/qfiles/out # ls -al |wc 38 335 4028
This indicates a serious backlog.
Feb 28 08:55:43 2016 (30307) <mesage-id> smtp to list for nnn recips, completed in t.ttt seconds
this is the postfix mail log?
No. Look at Mailman's smtp log. This may be /var/lib/mailman/logs/smtp or /usr/local/mailman/logs/smtp or /var/log/mailman/smtp or somewhere else depending on how mailman was installed
with each messages time stamp being t.ttt seconds later than the preceding message, the out/ queue is backlogged. If that is the case, Mailman isn't able to deliver fast enough to Postfix to keep up with it's volume.
Evidently... So what might be a fix? Because whatever it is it doesn't affect other mail. :(
Feb 28 17:17:18 2016 (29233) CAJS-itcfOi9J2dgX0U_+NR=RkywLeBgvgoE2z67TVL=oF71wUg@mail.gmail.com smtp to hangout for 10 recips, completed in 69.241 seconds Feb 28 17:18:27 2016 (29233) 56D30C67.7070300@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.239 seconds Feb 28 17:19:37 2016 (29233) 56D311F1.8080002@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.240 seconds Feb 28 17:20:46 2016 (29233) 56D336AA.7060709@crossroadstech.com smtp to hangout for 10 recips, completed in 69.247 seconds Feb 28 17:21:55 2016 (29233) 56D2F8D0.8080208@verizon.net smtp to hangout for 10 recips, completed in 69.229 seconds Feb 28 17:23:27 2016 (29233) 20160222074941.B47C9C193@s8.hostlocal.com smtp to hangout for 69 recips, completed in 91.821 seconds Feb 28 17:24:37 2016 (29233) 56CD2795.4020601@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.305 seconds Feb 28 17:25:46 2016 (29233) 56CE7EF5.6020208@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.235 seconds Feb 28 17:26:56 2016 (29233) 50797049-1456397806-cardhu_decombobulator_blackberry.rim.net-1607713858-@b1.c1.bise6.blackberry smtp to hangout for 10 recips, completed in 69.238 seconds Feb 28 17:28:05 2016 (29233) 20160226060443.GA18326@www.mrbrklyn.com smtp to hangout for 10 recips, completed in 69.249 seconds Feb 28 17:29:14 2016 (29233) 20160226073323.GD20244@linuxmafia.com smtp to hangout for 10 recips, completed in 69.227 seconds Feb 28 17:30:24 2016 (29233) E1aZLtt-00007s-Mj@crmserver1p.fsf.org smtp to hangout for 10 recips, completed in 69.236 seconds Feb 28 17:31:33 2016 (29233) 20160227045303.GH20244@linuxmafia.com smtp to hangout for 10 recips, completed in 69.364 seconds Feb 28 17:32:42 2016 (29233) 20160227080928.GH12323@linuxmafia.com smtp to hangout for 10 recips, completed in 69.198 seconds Feb 28 17:33:52 2016 (29233) 56D28F54.1050005@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.465 seconds Feb 28 17:35:02 2016 (29233) 20160228063512.GB14573@www.mrbrklyn.com smtp to hangout for 10 recips, completed in 69.229 seconds Feb 28 17:36:11 2016 (29233) CAAy8BASGqf4qkG61AbO_G-gj5QGNvzUCntSURRYMrvRkh0C2kA@mail.gmail.com smtp to hangout for 10 recips, completed in 69.301 seconds Feb 28 17:37:20 2016 (29233) CADJGPb-C-HMRGg-b5onMzywZaWdw=TF+iNBuAG6gXD7TVYMdpg@mail.gmail.com smtp to hangout for 10 recips, completed in 69.282 seconds Feb 28 17:38:30 2016 (29233) CADJGPb-C-HMRGg-b5onMzywZaWdw=TF+iNBuAG6gXD7TVYMdpg@mail.gmail.com smtp to hangout for 10 recips, completed in 69.288 seconds Feb 28 17:39:39 2016 (29233) CADJGPb-BUXSV=wuTyiy8oFgTuteH_Kpp4XFQKxoQv3zbKy4bVg@mail.gmail.com smtp to hangout for 10 recips, completed in 69.430 seconds Feb 28 17:40:49 2016 (29233) 329326E7-8ACC-4757-888E-098E498008C9@nyu.edu smtp to hangout for 10 recips, completed in 69.247 seconds Feb 28 17:41:58 2016 (29233) 20160228064203.GA14806@www.mrbrklyn.com smtp to hangout for 10 recips, completed in 69.225 seconds Feb 28 17:43:07 2016 (29233) 20160228071130.GA15677@www.mrbrklyn.com smtp to hangout for 10 recips, completed in 69.242 seconds Feb 28 17:44:17 2016 (29233) 56D2A094.7090306@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.230 seconds Feb 28 17:45:26 2016 (29233) 56D2A251.5070208@my.liu.edu smtp to hangout for 10 recips, completed in 69.223 seconds Feb 28 17:46:36 2016 (29233) 56D2A458.8020400@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.499 seconds Feb 28 17:47:45 2016 (29233) 20160228075600.GA16210@www.mrbrklyn.com smtp to hangout for 10 recips, completed in 69.254 seconds Feb 28 17:48:54 2016 (29233) 56D2B699.4000204@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.241 seconds Feb 28 17:50:04 2016 (29233) 56D2BA1C.9010002@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.190 seconds
-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com
Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013
On 02/28/2016 02:54 PM, Ruben Safir wrote:
Evidently... So what might be a fix? Because whatever it is it doesn't affect other mail. :(
Feb 28 17:17:18 2016 (29233) CAJS-itcfOi9J2dgX0U_+NR=RkywLeBgvgoE2z67TVL=oF71wUg@mail.gmail.com smtp to hangout for 10 recips, completed in 69.241 seconds Feb 28 17:18:27 2016 (29233) 56D30C67.7070300@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.239 seconds Feb 28 17:19:37 2016 (29233) 56D311F1.8080002@mrbrklyn.com smtp to hangout for 10 recips, completed in 69.240 seconds
...
So it takes Mailman over a minute to deliver to Postfix for 10 recipients. No matter what Mailman is doing, this is way too long by over a factor of 100.
Various things could be at the root of this, e.g. reject_unknown_recipient_domain in smtpd_recipient_restrictions particularly in combination with slow DNS lookups.
To say more, we'd need to see the output from 'postconf -n' and the contents of mm_cfg.py, but this is really a Postfix tuning issue, soo see http://www.postfix.org/TUNING_README.html and other Postfix resources as well. There is also information in our wiki, but a lot of the Mailman specific stuff is out of date, but go to http://wiki.list.org/FrontPage and search titles for "performance".
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (9)
-
Carl Zwanzig -
Hank van Cleef -
Jim Popovitch -
Keith Seyffarth -
Mark Sapiro -
Ruben Safir -
Seun Ojedeji -
Stephen J. Turnbull -
Tanstaafl