Hello,
I'm using Mailman in conjunction with Exim on FreeBSD 5.3 to support a small mailing list (60 members). All messages have been taking just over 105 seconds to deliver, according to /usr/local/mailman/log/smtp.
After searching through the archives to this list, I found that a garbage line in /etc/hosts was responsible for 75 seconds worth of that. Now we're down to consistently just over 30 seconds.
Where else should I look for things that could be causing this? Exim only has problems with mail routed through Mailman.
Herman
Looking for earth-friendly autos? Browse Top Cars by "Green Rating" at Yahoo! Autos' Green Center. http://autos.yahoo.com/green_center/
Herman Privyhum sent the message below at 15:14 3/8/2007:
This may just be the time needed to resolve the dns lookups for the outgoing mail.
You may want to consider using a caching dns resolver to cache the sddresses for some reasonable period. Getting the resolved addresses out of cache will be much faster than attempting to resolve them over the network every time.
Then again, I could be completely off here and something else is causing the delay. I don't know Exim at all and sendmail only in the barest manner so I cannot be much help beyond this.
Dragon
Venimus, Saltavimus, Bibimus (et naribus canium capti sumus)
The esteemed Dragon has said:
Maybe this is a good time to ask just how DNS-intensive the non-sendmail MTA's are. I am finishing off the basics on installing sendmail with Mailman, and am including some discussion of the need to install a good fast-response caching DNS server to work with sendmail.
There is very little discussion in the sendmail literature about DNS, except for an acknowledgement that sendmail uses DNS intensively if it's in the "hosts" line in nsswitch.conf (Solaris name).
I have to confess that I was a bit slow on the uptake to install local DNS on my systems. The folks at upstream feed, whose DNS servers I was using, said, "you do plan to install local DNS, don't you," with a certain pointedness. I dawdled until I could get the 5th edition of Liu and Albitz "DNS and BIND" (O'Reilly, 2006), which has a chapter on DNS and SMTP mail, and after reading through sat down to install a caching server immediately.
The results were just plain startling. I wish I'd done this five years ago.
Since then I've installed master and slave servers for my Intranet LAN, but I would heartly recommend having at least a plain caching server on the box that's running the MTA.
While all of my experience is with sendmail, I'm inclined to suspect that the other MTA's all can stand a shot of local DNS service. Anybody who can confirm this for Postfix, Exim, etc.?
Hank
At 8:46 PM -0700 3/8/07, vancleef@lostwells.net wrote:
All MTAs I know of are pretty DNS-intensive in their operation. The more anti-spam or anti-virus filtering you do, or the more other things you do to check the incoming mail, the more DNS-intensive that work is going to be.
Of course, most MTAs should give you options on how to configure them so that they don't generate any DNS traffic at all, but then what you're doing is effectively turning off about 99.99% of what the MTA is intended to do when handling mail.
In this respect, I don't think that sendmail is necessarily much worse or much better than any other MTA.
Years ago, this was actually a bit of a sore point amongst the experts. Some said that you were better off having a smaller number of centralized caching nameservers, which handled all DNS traffic for the entire network.
Others said that you're better off having caching nameservers running on each box, to spread that load out.
Of course, the issue there is that Box A might do a DNS query of some sort, and retrieve data that could later be used by Box B, but if both machines are running their own nameservers as opposed to a centralized caching nameserver, then both machines will end up doing the same query, causing increased load on the remote end, etc....
Moreover, large caching nameservers can take up hundreds of megabytes (or even a couple of gigabytes) of RAM, so if you've got servers that are already using lots of RAM to process all their "real" work, then you may not have enough RAM to also run a large caching nameserver on the box.
Finally, sometimes consistency is more important than raw speed. In other words, sometimes it's more important that the clients see that they get the same answers regardless of which server they ask, and the actual raw performance is not quite so important. For example, when an AOL user sends e-mail to a remote recipient, it would be really bad for that user to get "okay, message accepted" on the first try and then "invalid domain" on the second try, and then get "okay, message accepted" on the third try, or whatever. Since the DNS changes frequently, you could easily wind up with some pretty radically different views of the world on different servers, based on when they asked what questions.
To solve all these issues, what was recommended was a hybrid approach. Run local caching-only servers on each box, but then have them forward all outgoing queries to a central set of caching-only nameservers.
The local nameserver would short-circuit all the repetitive queries from the same application to talk to the same remote system, while the centralized caching nameservers would ensure that everyone gets the same answer to a particular question, and would ensure that you don't actually send your queries to the outside world unless no machine at that site had asked that question within the time-to-live of the answer.
DNS experts now agree that it's a generally a bad idea to have hierarchies of nameservers, although the overall problems have not otherwise changed.
So, pick your poison, but don't try to go with the hybrid approach. It creates too much of a central bottleneck and slows things down, and it also reduces your overall reliability of the system.
Of course, all detailed DNS questions should be asked on the appropriate mailing lists and/or newsgroups, although I can try to summarize as best I can -- I was a technical reviewer of 2nd edition of Cricket's book, and I'm in the process of writing my own book on DNS security.
All MTAs I know of make intensive use of the DNS -- sendmail, postfix, Exim, etc....
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
The esteemed Brad Knowles has said:
Thanks for the confirmation of what I'd suspected. Since I've lived with sendmail "ever since there was a sendmail" (not true, I think more like "since 1988"), I haven't tried to work with the other MTA's.
The need for a quick, robust, and reliable name service is in the nature of the beast, particularly when supporting a mail list server, where one incoming mail goes out to a thousand (if not "thousands") addresses.
I'm not going to try to address LDAP, NIS or NIS+. It takes about ten minutes to set up the tables for named for a caching server. That is what I would recommend for the shop that's currenly using /etc/hosts and remote (i.e. "somebody else's") DNS servers for everything external.
I'm going to snip this discussion of how to configure DNS on a site here. For one thing, I think that we need to keep the focus on Mailman, and simply answer the question "do you need local name service with Mailman?" with "Yes."
For another, Cricket Liu and Paul Albitz, in the current 5th edition (2006) of "DNS and BIND" have done a very good job of discussing far more of the ins and outs of configuring DNS in a 600-page book. They devote an entire chapter to DNS with Unix mail.
For Mailman support purposes, for both sendmail and DNS, I think we need to focus on a "keep it simple" approach. Both my site and the former Mailman host site for my primary list are "one worker, one box, one routable IP" sites, where Mailman, Apache, sendmail, and named all get handled on a single node. I also administer another ISP site 400 miles away (along with five others) that has separate boxes for login server, mail server, web server, news server, NFS RAID server, and two DNS servers, all with their own routable IP's. We did put Mailman on that site as well. Suffice it to say that there are a few differences in sendmail, apache, and named configurations between those two sites, none of which affect Mailman.
So I'd have to say, Mailman with sendmail and BIND DNS, yes, very easy to do. Follow Barry Warsaw's excellent Mailman build guide, go through four steps to configure sendmail (I'm about to send this off to the list), configure your DNS per Liu and Albitz, and go. Needless to say, doing this on an earthlink-sized system and doing it on a one-box oldfogey.net system aren't the same in many ways that are quite separate from Mailman considerations.
Hank
At 3:14 PM -0800 3/8/07, Herman Privyhum wrote:
Look at your MTA logs. They should show you when a given message comes in for a given user, and when that is successfully transmitted. I'd be willing to bet you're waiting on DNS timeouts at the remote end for one or more of your users -- their MTA is slowing you down, maybe as a result of trying to do a reverse DNS lookup on your IP address.
If you haven't already enabled it, try turning on full personalization, so that you send a unique copy of the message to each recipient. For each recipient, look at the date/time stamp of when the message goes into Exim and when Exim says that the remote end has accepted the message. That should tell you where things are slowing down.
Maybe it's not a DNS timeout, maybe it's a timeout of some other sort. Maybe it's a result of doing anti-spam/anti-virus processing on inbound as well as outbound e-mail. Maybe their mail servers are just very slow. But at least you should know where the slowdown is coming from.
You may also be able to turn on some increased parallelism in the delivery process, so that other recipients can be handled while you're waiting on the slower sites to respond.
But I wouldn't feel bad at getting everything out in 30 seconds.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
--- Brad Knowles <brad@shub-internet.org> wrote:
Thanks to all for the thorough replies. It appears that the solution actually lies in disabling ident.
In Exim, this is achieved by setting the timeout to 0.
#rfc1413_query_timeout = 30s
rfc1413_query_timeout = 0sI may go back and turn it on again with a smaller timeout (suggestions as to how long is reasonable?).
Before:
Mar 08 15:18:33 2007 (35218) <p0624081fc2163bb796bf@OBSCURED> smtp for 60 recips, completed in 211.309 seconds
After:
Mar 08 19:48:40 2007 (553) <008301c761fd$9d43e0c0$6704010a@OBSCURED> smtp for 61 recips, completed in 1.792 seconds
Cheers,
Herman
Food fight? Enjoy some healthy debate in the Yahoo! Answers Food & Drink Q&A. http://answers.yahoo.com/dir/?link=list&sid=396545367
At 9:44 PM -0800 3/8/07, Herman Privyhum wrote:
Thanks to all for the thorough replies. It appears that the solution actually lies in disabling ident.
Ahh, yes. You should definitely disable IDENT. I didn't know that any modern MTAs actually used it.
I may go back and turn it on again with a smaller timeout (suggestions as to how long is reasonable?).
Just leave it turned off. Using it means that you trust the other end to not lie to you, and on the modern Internet you can't trust the other end to do much of anything. You have to assume that all remote machines will always lie to you, and therefore you can't trust anything they send you.
At the very least, if you're going to use it, make sure you only use it on your local network of machines that you control. On those machines, you could use a lower timeout, such as ten or fifteen seconds.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
--- Brad Knowles <brad@shub-internet.org> wrote:
Here's Philip Hazel's rationale:
http://xrl.us/u8pf (Link to www.exim.org)
Herman
Now that's room service! Choose from over 150,000 hotels in 45,000 destinations on Yahoo! Travel to find your fit. http://farechase.yahoo.com/promo-generic-14795097
At 10:16 PM -0800 3/8/07, Herman Privyhum wrote:
http://xrl.us/u8pf (Link to www.exim.org)
So Phil says that he runs a trustworthy IDENT server on his box. Fine. But plenty of spammers, phishers, and other nefarious types out there will try to use IDENT as another vector to exploit for use in breaking into your system, or for tricking you into believing whatever lies they want you to believe.
Unfortunately, there isn't a trustworthy system to tell you which sites run trustworthy IDENT servers. So, you've got to decide what the relative risks and values are.
Moreover, damn few sites run multiuser systems like that anymore. With NAT, you could hide hundreds or millions of machines behind a single IP address, and IDENT to the NAT/firewall box would be pretty meaningless.
Sorry. IDENT was useful fifteen or twenty years ago, but there are far too many holes in that technique these days.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
At 1:04 AM -0600 3/9/07, Brad Knowles quoted Herman Privyhum:
http://xrl.us/u8pf (Link to www.exim.org)
So Phil says that he runs a trustworthy IDENT server on his box.
BTW, that article is eight years old now. Eight years in "real" time is over five Internet generations. We were still working on Web 0.9 at the time, much less Web 2.0.
RFC 2780 was published in March of 2000. RFC 4856 was just recently published, in February of 2007. That's 2076 RFCs published since Phil's article was originally posted.
Thanks mostly to the Hobbes Internet Timeline (see <http://www.zakon.org/robert/internet/timeline/>), we know that 2000 was the year that Internet2 backbone network deployed IPv6. The year that Mexico finally got a fully operational connection to Internet2. The year that the French court ruled Yahoo! must block French users from accessing hate memorabilia in its auction site. Technologies of the year were ASP, Napster, and DeCSS.
2001 was the year of Grid Computing and P2P. 2002 was the year of the FBI teaming up with Terra Lycos to disseminate virtual wanted posts across the Web portal's properties -- does anyone even remember Terra Lycos anymore? 2003 was the year that PIR took over as the .org registry operator, the Recording Industry Association of America (RIAA) sued 261 individuals on 8 Sep for allegedly distributing copyright music files over peer-to-peer networks, and VeriSign deployed a wildcard service (Site Finder) into the .com and .net TLDs causing much confusion as URLs with invalid domains are redirected to a VeriSign page. 2004 was the year that Network Solutions began offering 100 year domain registration, VeriSign Naming and Directory Service (VNDS) began updating all 13 .com/.net authoritative name servers in near real-time vs. twice each day, and the Internet Worm called MyDoom (or Novarg), spread through Internet servers. 2005 was the year that YouTube.com was launched.
Estimates of the size of the Internet, by year:
01/00 72,398,092
07/00 93,047,785
01/01 109,574,429
07/01 125,888,197
01/02 147,344,723
07/02 162,128,493
01/03 171,638,297
01/04 233,101,481
07/04 285,139,107
01/05 317,646,084
07/05 353,284,187
01/06 394,991,609
07/06 439,286,364Growth factor = 439286364/72398092 = 6.06765
According to the wikipedia page at <http://en.wikipedia.org/wiki/Phishing>, the first known example of Phishing outside of AOL didn't occur until June of 2001. I quote:
By 2004, phishing was recognized as fully industrialized, in
the sense of an economy of crime: specializations emerged on
a global scale and provided components for cash which were
assembled into a finished attack.I'm sorry. I don't see how Phil's views from eight years ago on this subject are relevant to how computer systems should be operated in this modern world.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
--- Brad Knowles <brad@shub-internet.org> wrote:
Thanks for the in-depth commentary.
Here's the modern FAQ entry, FWIW:
<http://www.exim.org/eximwiki/FAQ/Miscellaneous/Q5023>
(The reasoning does not differ substantially from that in Hazel's seven-year-old email.)
If disabling IDENT is so crucial, as both you and my experience with a triflingly small list seem to argue, it must be important to amend the Mailman-Exim Howto, no?
Found here: http://www.exim.org/howto/mailman21.html
Linked from here: http://www.gnu.org/software/mailman/docs.html
Herman
8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut. http://tools.search.yahoo.com/shortcuts/#news
At 5:15 AM -0800 3/9/07, Herman Privyhum wrote:
Well, it's their HowTo, so I don't see that this is something that we should (or could) be updating ourselves.
That said, it might be a good idea to put a note in our FAQ Wizard warning people about the risks of IDENT. This isn't really a Mailman thing per se, more of an MTA issue, but it is still something that people running Mailman servers might run into.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
On Fri, 9 Mar 2007, Brad Knowles wrote:
This is a common misconception of what IDENT is/was for. IDENT was not intended to provide reliable authentication, as to who owned a connection. Rather, IDENT was a way of providing information such that a sysadmin could figure out later which of their own users had done something bad, or had their account compromised.
People then started using it this way, possibly due to the inclusion in tcp wrappers, but as I recall it wasn't the original purpose.
In other words, as the recipient I have no reason to trust the string. But if I am on the reciving end of an attack from a multi-user machine and am reporting it to the owner of the machine, I would give them the IDENT data I capture so they can better track what happened on their machine. And even on a non-multi user machine, it could help narrow down what process was compromised.
There was at least one IDENT server that would return a seemingly random string, that could be decrypted by the sysadmin to know what the account was without divulging the actual name to the outside.
========================================================== Chris Candreva -- chris@westnet.com -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/
At 9:48 AM -0500 3/9/07, Christopher X. Candreva wrote:
This is a common misconception of what IDENT is/was for. IDENT was not intended to provide reliable authentication, as to who owned a connection.
I know what the purpose of IDENT is. I wrote the original sendmail FAQ entry on this subject back in 1995.
The problem is that while IDENT can be useful in some cases, the risks and costs of IDENT on the modern Internet is so high that it's not worth the effort.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
On Sat, 10 Mar 2007, Brad Knowles wrote:
This is a common misconception of what IDENT is/was for. IDENT was not
I know what the purpose of IDENT is. I wrote the original sendmail FAQ entry on this subject back in 1995.
Sorry, automatic response of the fingers, usually reserved for IRC operators who somehow think accepting connections only from hosts running IDENT does something for them.
In fact it's so automatic I have the funny feeling I'm commited this same faux pass with you before.
-Chris
========================================================== Chris Candreva -- chris@westnet.com -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/
On 3/8/07 10:44 PM, "Herman Privyhum" <privyhum@yahoo.com> wrote:
Just a note: The default value of rfc1413_query_timeout was reduced in recent versions of Exim to, I think, 5s.
That was in preference to changing it to 0s, as a change to 0s would impact sites which don't change that default, something Philip doesn't like to do except when the major version changes.
Philip will be retired (late September) before Exim 5 is out--the Exim community is well along in setting up a structure to deal with that. His retirement is, IMHO, much scarier with respect to the PCRE library than is is with respect to Exim.
--John
Herman Privyhum wrote:
Keep in mind that this SMTP transaction with Exim has 60 RCPT TO commands and responses which is probably where most of the time is spent. If Exim is doing DNS verifies on the recipient domains during the incoming SMTP, 0.5 sec per recipient may not be that bad.
The solution is to not do DNS verification on mail from localhost. See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.011.htp>.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Herman Privyhum sent the message below at 15:14 3/8/2007:
This may just be the time needed to resolve the dns lookups for the outgoing mail.
You may want to consider using a caching dns resolver to cache the sddresses for some reasonable period. Getting the resolved addresses out of cache will be much faster than attempting to resolve them over the network every time.
Then again, I could be completely off here and something else is causing the delay. I don't know Exim at all and sendmail only in the barest manner so I cannot be much help beyond this.
Dragon
Venimus, Saltavimus, Bibimus (et naribus canium capti sumus)
The esteemed Dragon has said:
Maybe this is a good time to ask just how DNS-intensive the non-sendmail MTA's are. I am finishing off the basics on installing sendmail with Mailman, and am including some discussion of the need to install a good fast-response caching DNS server to work with sendmail.
There is very little discussion in the sendmail literature about DNS, except for an acknowledgement that sendmail uses DNS intensively if it's in the "hosts" line in nsswitch.conf (Solaris name).
I have to confess that I was a bit slow on the uptake to install local DNS on my systems. The folks at upstream feed, whose DNS servers I was using, said, "you do plan to install local DNS, don't you," with a certain pointedness. I dawdled until I could get the 5th edition of Liu and Albitz "DNS and BIND" (O'Reilly, 2006), which has a chapter on DNS and SMTP mail, and after reading through sat down to install a caching server immediately.
The results were just plain startling. I wish I'd done this five years ago.
Since then I've installed master and slave servers for my Intranet LAN, but I would heartly recommend having at least a plain caching server on the box that's running the MTA.
While all of my experience is with sendmail, I'm inclined to suspect that the other MTA's all can stand a shot of local DNS service. Anybody who can confirm this for Postfix, Exim, etc.?
Hank
At 8:46 PM -0700 3/8/07, vancleef@lostwells.net wrote:
All MTAs I know of are pretty DNS-intensive in their operation. The more anti-spam or anti-virus filtering you do, or the more other things you do to check the incoming mail, the more DNS-intensive that work is going to be.
Of course, most MTAs should give you options on how to configure them so that they don't generate any DNS traffic at all, but then what you're doing is effectively turning off about 99.99% of what the MTA is intended to do when handling mail.
In this respect, I don't think that sendmail is necessarily much worse or much better than any other MTA.
Years ago, this was actually a bit of a sore point amongst the experts. Some said that you were better off having a smaller number of centralized caching nameservers, which handled all DNS traffic for the entire network.
Others said that you're better off having caching nameservers running on each box, to spread that load out.
Of course, the issue there is that Box A might do a DNS query of some sort, and retrieve data that could later be used by Box B, but if both machines are running their own nameservers as opposed to a centralized caching nameserver, then both machines will end up doing the same query, causing increased load on the remote end, etc....
Moreover, large caching nameservers can take up hundreds of megabytes (or even a couple of gigabytes) of RAM, so if you've got servers that are already using lots of RAM to process all their "real" work, then you may not have enough RAM to also run a large caching nameserver on the box.
Finally, sometimes consistency is more important than raw speed. In other words, sometimes it's more important that the clients see that they get the same answers regardless of which server they ask, and the actual raw performance is not quite so important. For example, when an AOL user sends e-mail to a remote recipient, it would be really bad for that user to get "okay, message accepted" on the first try and then "invalid domain" on the second try, and then get "okay, message accepted" on the third try, or whatever. Since the DNS changes frequently, you could easily wind up with some pretty radically different views of the world on different servers, based on when they asked what questions.
To solve all these issues, what was recommended was a hybrid approach. Run local caching-only servers on each box, but then have them forward all outgoing queries to a central set of caching-only nameservers.
The local nameserver would short-circuit all the repetitive queries from the same application to talk to the same remote system, while the centralized caching nameservers would ensure that everyone gets the same answer to a particular question, and would ensure that you don't actually send your queries to the outside world unless no machine at that site had asked that question within the time-to-live of the answer.
DNS experts now agree that it's a generally a bad idea to have hierarchies of nameservers, although the overall problems have not otherwise changed.
So, pick your poison, but don't try to go with the hybrid approach. It creates too much of a central bottleneck and slows things down, and it also reduces your overall reliability of the system.
Of course, all detailed DNS questions should be asked on the appropriate mailing lists and/or newsgroups, although I can try to summarize as best I can -- I was a technical reviewer of 2nd edition of Cricket's book, and I'm in the process of writing my own book on DNS security.
All MTAs I know of make intensive use of the DNS -- sendmail, postfix, Exim, etc....
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
The esteemed Brad Knowles has said:
Thanks for the confirmation of what I'd suspected. Since I've lived with sendmail "ever since there was a sendmail" (not true, I think more like "since 1988"), I haven't tried to work with the other MTA's.
The need for a quick, robust, and reliable name service is in the nature of the beast, particularly when supporting a mail list server, where one incoming mail goes out to a thousand (if not "thousands") addresses.
I'm not going to try to address LDAP, NIS or NIS+. It takes about ten minutes to set up the tables for named for a caching server. That is what I would recommend for the shop that's currenly using /etc/hosts and remote (i.e. "somebody else's") DNS servers for everything external.
I'm going to snip this discussion of how to configure DNS on a site here. For one thing, I think that we need to keep the focus on Mailman, and simply answer the question "do you need local name service with Mailman?" with "Yes."
For another, Cricket Liu and Paul Albitz, in the current 5th edition (2006) of "DNS and BIND" have done a very good job of discussing far more of the ins and outs of configuring DNS in a 600-page book. They devote an entire chapter to DNS with Unix mail.
For Mailman support purposes, for both sendmail and DNS, I think we need to focus on a "keep it simple" approach. Both my site and the former Mailman host site for my primary list are "one worker, one box, one routable IP" sites, where Mailman, Apache, sendmail, and named all get handled on a single node. I also administer another ISP site 400 miles away (along with five others) that has separate boxes for login server, mail server, web server, news server, NFS RAID server, and two DNS servers, all with their own routable IP's. We did put Mailman on that site as well. Suffice it to say that there are a few differences in sendmail, apache, and named configurations between those two sites, none of which affect Mailman.
So I'd have to say, Mailman with sendmail and BIND DNS, yes, very easy to do. Follow Barry Warsaw's excellent Mailman build guide, go through four steps to configure sendmail (I'm about to send this off to the list), configure your DNS per Liu and Albitz, and go. Needless to say, doing this on an earthlink-sized system and doing it on a one-box oldfogey.net system aren't the same in many ways that are quite separate from Mailman considerations.
Hank
At 3:14 PM -0800 3/8/07, Herman Privyhum wrote:
Look at your MTA logs. They should show you when a given message comes in for a given user, and when that is successfully transmitted. I'd be willing to bet you're waiting on DNS timeouts at the remote end for one or more of your users -- their MTA is slowing you down, maybe as a result of trying to do a reverse DNS lookup on your IP address.
If you haven't already enabled it, try turning on full personalization, so that you send a unique copy of the message to each recipient. For each recipient, look at the date/time stamp of when the message goes into Exim and when Exim says that the remote end has accepted the message. That should tell you where things are slowing down.
Maybe it's not a DNS timeout, maybe it's a timeout of some other sort. Maybe it's a result of doing anti-spam/anti-virus processing on inbound as well as outbound e-mail. Maybe their mail servers are just very slow. But at least you should know where the slowdown is coming from.
You may also be able to turn on some increased parallelism in the delivery process, so that other recipients can be handled while you're waiting on the slower sites to respond.
But I wouldn't feel bad at getting everything out in 30 seconds.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
--- Brad Knowles <brad@shub-internet.org> wrote:
Thanks to all for the thorough replies. It appears that the solution actually lies in disabling ident.
In Exim, this is achieved by setting the timeout to 0.
#rfc1413_query_timeout = 30s
rfc1413_query_timeout = 0sI may go back and turn it on again with a smaller timeout (suggestions as to how long is reasonable?).
Before:
Mar 08 15:18:33 2007 (35218) <p0624081fc2163bb796bf@OBSCURED> smtp for 60 recips, completed in 211.309 seconds
After:
Mar 08 19:48:40 2007 (553) <008301c761fd$9d43e0c0$6704010a@OBSCURED> smtp for 61 recips, completed in 1.792 seconds
Cheers,
Herman
Food fight? Enjoy some healthy debate in the Yahoo! Answers Food & Drink Q&A. http://answers.yahoo.com/dir/?link=list&sid=396545367
At 9:44 PM -0800 3/8/07, Herman Privyhum wrote:
Thanks to all for the thorough replies. It appears that the solution actually lies in disabling ident.
Ahh, yes. You should definitely disable IDENT. I didn't know that any modern MTAs actually used it.
I may go back and turn it on again with a smaller timeout (suggestions as to how long is reasonable?).
Just leave it turned off. Using it means that you trust the other end to not lie to you, and on the modern Internet you can't trust the other end to do much of anything. You have to assume that all remote machines will always lie to you, and therefore you can't trust anything they send you.
At the very least, if you're going to use it, make sure you only use it on your local network of machines that you control. On those machines, you could use a lower timeout, such as ten or fifteen seconds.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
--- Brad Knowles <brad@shub-internet.org> wrote:
Here's Philip Hazel's rationale:
http://xrl.us/u8pf (Link to www.exim.org)
Herman
Now that's room service! Choose from over 150,000 hotels in 45,000 destinations on Yahoo! Travel to find your fit. http://farechase.yahoo.com/promo-generic-14795097
At 10:16 PM -0800 3/8/07, Herman Privyhum wrote:
http://xrl.us/u8pf (Link to www.exim.org)
So Phil says that he runs a trustworthy IDENT server on his box. Fine. But plenty of spammers, phishers, and other nefarious types out there will try to use IDENT as another vector to exploit for use in breaking into your system, or for tricking you into believing whatever lies they want you to believe.
Unfortunately, there isn't a trustworthy system to tell you which sites run trustworthy IDENT servers. So, you've got to decide what the relative risks and values are.
Moreover, damn few sites run multiuser systems like that anymore. With NAT, you could hide hundreds or millions of machines behind a single IP address, and IDENT to the NAT/firewall box would be pretty meaningless.
Sorry. IDENT was useful fifteen or twenty years ago, but there are far too many holes in that technique these days.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
At 1:04 AM -0600 3/9/07, Brad Knowles quoted Herman Privyhum:
http://xrl.us/u8pf (Link to www.exim.org)
So Phil says that he runs a trustworthy IDENT server on his box.
BTW, that article is eight years old now. Eight years in "real" time is over five Internet generations. We were still working on Web 0.9 at the time, much less Web 2.0.
RFC 2780 was published in March of 2000. RFC 4856 was just recently published, in February of 2007. That's 2076 RFCs published since Phil's article was originally posted.
Thanks mostly to the Hobbes Internet Timeline (see <http://www.zakon.org/robert/internet/timeline/>), we know that 2000 was the year that Internet2 backbone network deployed IPv6. The year that Mexico finally got a fully operational connection to Internet2. The year that the French court ruled Yahoo! must block French users from accessing hate memorabilia in its auction site. Technologies of the year were ASP, Napster, and DeCSS.
2001 was the year of Grid Computing and P2P. 2002 was the year of the FBI teaming up with Terra Lycos to disseminate virtual wanted posts across the Web portal's properties -- does anyone even remember Terra Lycos anymore? 2003 was the year that PIR took over as the .org registry operator, the Recording Industry Association of America (RIAA) sued 261 individuals on 8 Sep for allegedly distributing copyright music files over peer-to-peer networks, and VeriSign deployed a wildcard service (Site Finder) into the .com and .net TLDs causing much confusion as URLs with invalid domains are redirected to a VeriSign page. 2004 was the year that Network Solutions began offering 100 year domain registration, VeriSign Naming and Directory Service (VNDS) began updating all 13 .com/.net authoritative name servers in near real-time vs. twice each day, and the Internet Worm called MyDoom (or Novarg), spread through Internet servers. 2005 was the year that YouTube.com was launched.
Estimates of the size of the Internet, by year:
01/00 72,398,092
07/00 93,047,785
01/01 109,574,429
07/01 125,888,197
01/02 147,344,723
07/02 162,128,493
01/03 171,638,297
01/04 233,101,481
07/04 285,139,107
01/05 317,646,084
07/05 353,284,187
01/06 394,991,609
07/06 439,286,364Growth factor = 439286364/72398092 = 6.06765
According to the wikipedia page at <http://en.wikipedia.org/wiki/Phishing>, the first known example of Phishing outside of AOL didn't occur until June of 2001. I quote:
By 2004, phishing was recognized as fully industrialized, in
the sense of an economy of crime: specializations emerged on
a global scale and provided components for cash which were
assembled into a finished attack.I'm sorry. I don't see how Phil's views from eight years ago on this subject are relevant to how computer systems should be operated in this modern world.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
--- Brad Knowles <brad@shub-internet.org> wrote:
Thanks for the in-depth commentary.
Here's the modern FAQ entry, FWIW:
<http://www.exim.org/eximwiki/FAQ/Miscellaneous/Q5023>
(The reasoning does not differ substantially from that in Hazel's seven-year-old email.)
If disabling IDENT is so crucial, as both you and my experience with a triflingly small list seem to argue, it must be important to amend the Mailman-Exim Howto, no?
Found here: http://www.exim.org/howto/mailman21.html
Linked from here: http://www.gnu.org/software/mailman/docs.html
Herman
8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut. http://tools.search.yahoo.com/shortcuts/#news
At 5:15 AM -0800 3/9/07, Herman Privyhum wrote:
Well, it's their HowTo, so I don't see that this is something that we should (or could) be updating ourselves.
That said, it might be a good idea to put a note in our FAQ Wizard warning people about the risks of IDENT. This isn't really a Mailman thing per se, more of an MTA issue, but it is still something that people running Mailman servers might run into.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
On Fri, 9 Mar 2007, Brad Knowles wrote:
This is a common misconception of what IDENT is/was for. IDENT was not intended to provide reliable authentication, as to who owned a connection. Rather, IDENT was a way of providing information such that a sysadmin could figure out later which of their own users had done something bad, or had their account compromised.
People then started using it this way, possibly due to the inclusion in tcp wrappers, but as I recall it wasn't the original purpose.
In other words, as the recipient I have no reason to trust the string. But if I am on the reciving end of an attack from a multi-user machine and am reporting it to the owner of the machine, I would give them the IDENT data I capture so they can better track what happened on their machine. And even on a non-multi user machine, it could help narrow down what process was compromised.
There was at least one IDENT server that would return a seemingly random string, that could be decrypted by the sysadmin to know what the account was without divulging the actual name to the outside.
========================================================== Chris Candreva -- chris@westnet.com -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/
At 9:48 AM -0500 3/9/07, Christopher X. Candreva wrote:
This is a common misconception of what IDENT is/was for. IDENT was not intended to provide reliable authentication, as to who owned a connection.
I know what the purpose of IDENT is. I wrote the original sendmail FAQ entry on this subject back in 1995.
The problem is that while IDENT can be useful in some cases, the risks and costs of IDENT on the modern Internet is so high that it's not worth the effort.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
On Sat, 10 Mar 2007, Brad Knowles wrote:
This is a common misconception of what IDENT is/was for. IDENT was not
I know what the purpose of IDENT is. I wrote the original sendmail FAQ entry on this subject back in 1995.
Sorry, automatic response of the fingers, usually reserved for IRC operators who somehow think accepting connections only from hosts running IDENT does something for them.
In fact it's so automatic I have the funny feeling I'm commited this same faux pass with you before.
-Chris
========================================================== Chris Candreva -- chris@westnet.com -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/
On 3/8/07 10:44 PM, "Herman Privyhum" <privyhum@yahoo.com> wrote:
Just a note: The default value of rfc1413_query_timeout was reduced in recent versions of Exim to, I think, 5s.
That was in preference to changing it to 0s, as a change to 0s would impact sites which don't change that default, something Philip doesn't like to do except when the major version changes.
Philip will be retired (late September) before Exim 5 is out--the Exim community is well along in setting up a structure to deal with that. His retirement is, IMHO, much scarier with respect to the PCRE library than is is with respect to Exim.
--John
Herman Privyhum wrote:
Keep in mind that this SMTP transaction with Exim has 60 RCPT TO commands and responses which is probably where most of the time is spent. If Exim is doing DNS verifies on the recipient domains during the incoming SMTP, 0.5 sec per recipient may not be that bad.
The solution is to not do DNS verification on mail from localhost. See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.011.htp>.
-- Mark Sapiro <msapiro@value.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (7)
-
Brad Knowles -
Christopher X. Candreva -
Dragon -
Herman Privyhum -
John W. Baxter -
Mark Sapiro -
vancleef＠lostwells.net