[Mailman-Users] MTAs

[Brad Knowles]

At 2:25 PM -0800 2005-03-25, Heather Madrone wrote:

>  FreeBSD works great if you don't need a keyboard, a mouse, or a monitor.

	They are oriented towards the serial console at the moment, but 
I'm sure that the rest will come along.  I've got four UltraSPARC 10 
clones that I plan on using under FreeBSD (perhaps turning them into 
a small package build cluster), and I'll make myself a guinea pig.

>  Not an insurmountable problem for a server, but the FreeBSD hardware
>  support of the sparcs is a lot spottier than NetBSD or OpenBSD.

	It's not as good for FreeBSD as it should be, no.

>  All mailing lists on the server silently stop sending posts.  No 
>errors in the
>  logs, no console messages, no nothing.  Qrunners still appear to be
>  functioning normally.  A mail loopback test diagnoses the problem and
>  restarting Mailman clears it.

	Huh.  Weird.  I've never heard of anything like that.

>  With exim, trying to do VERP processing on the digest runs caused Mailman
>  to flood exim, even when I turned the number of sessions and connections
>  way down.  It also brought my poor Mac, which was not meant to be a server,
>  to its knees.  I'm hoping that the sparc will be able to handle peak
>  loads more gracefully than a Powerbook.

	Depends on the powerbook, the UltraSPARC configuration, etc....


	Generally speaking, for things like this, assuming you have 
enough RAM and network I/O capacity, the principal limiting factor 
becomes synchronous meta-data I/O.  Every time you create a file, 
delete a file, move a file from one directory to another, or rename a 
file, you're doing synchronous meta-data I/O.  This means that the 
entire directory has to be locked while the operation is taking place.

	Now, locking the entire directory, doing the operationg, and 
unlocking the entire directory, is something that usually happens 
pretty quickly.  However, try doing that thousands of times per 
second, and you run into very serious bottleneck issues.

	Of course, one thing an MTA does more than anything else is 
create lots and lots of little files, which don't live for very long.


	In this situation, *BSD with softupdates will be your best bet on 
the filesystem side.  The cool thing about softupdates is that it 
re-orders the disk I/O operations in a safe manner, and if the file 
is created and goes away quickly enough, then the I/O is never pushed 
to disk at all.

	All the *BSD implementations should be capable of enabling softupdates.

	If you do end up having to go with Solaris, make sure that you 
enable filesystem logging.  It's not as good as softupdates (although 
Sun did buy the rights to softupdates from Kirk, and is incorporating 
them into a future version of Solaris), but filesystem logging is 
better than nothing.

	Note that Apple doesn't have anything like softupdates for MacOS 
X, although more recent versions have introduced a type of journaling 
for the filesystem.  Unfortunately, in this case it actually slows 
things down a bit, but it does improve robustness in a crash, so 
overall it's still a win.

>  One of the reasons that I'm considering Postfix is that I wonder whether
>  it might be more efficient than exim.  I have a hunch that exim might do
>  more work than it needs to to get the job done.

	In all my testing, when it comes to the basic operations of a 
mail server, postfix has been the most efficient MTA that I've ever 
found.  I've tried very, very hard to push sendmail to be faster and 
more efficient, and I still believe that can be done, although I 
haven't managed it -- and I consider myself to be a specialist in 
this area.

	I haven't been able to do much testing with Exim towards this 
end.  However, Phil Hazel has said some very complimentary things 
about postfix in this regard, and I have a very hard time believing 
that Exim can compete.

>  Okay, so it's not simple.  Easy enough to figure out, though, and it
>  does give you the ability to tune exim to deal with the kinds of errors
>  that you run into.  It pains me to see endless retries of errors that
>  aren't going to go away in minutes.

	Postfix tries to be smarter about these sorts of things without 
requiring any kind of complex configuration.  It uses a bounded 
exponential backoff scheme, the same type of scheme that has been 
proven over time throughout TCP/IP, Ethernet, and other network 
protocols.

	Postfix does include some pretty advanced tools for diagnosing 
queueing problems (see <http://www.postfix.org/QSHAPE_README.html>), 
and once you've identified any particular problem domains you can 
then create separate queues dedicated to them so that everything else 
is unaffected.  There's also some pretty extensive performance tuning 
tips at <http://www.postfix.org/TUNING_README.html>.

>  How would you rate Postfix's efficiency?  Is it fast?  Does it limit
>  network traffic to the necessary and keep its disk reads and writes
>  down to a dull roar?

	Postfix is the fastest MTA I've ever tested.  It's smart about 
trying parallelism to remote sites, and using backing off when it 
detects that the remote site can't handle the load.  If you like, 
you're able to configure higher amounts of parallelism for given 
remote sites.

	Pretty much everything in postfix is table driven, and those 
tables can be in most any form of database you like -- Berkeley DB, 
*dbm, MySQL, LDAP, plain text files, or whatever.  In some areas, it 
uses regular expressions, and you typically have options of 
Unix-style regular expressions or Perl-compatible REs, etc....

>  Bandwidth is a consideration, since this will all be flowing on a fairly
>  slow DSL line (speed in this case limited by geography).

	Personally, I can't think of an MTA that would be better suited 
for this environment, but then I have been involved in postfix since 
'97, back before it was officially announced and it was still called 
VMailer.  I'm a bit biased.  ;)

>  Do you know why the choice was made to use Postfix on any of those
>  sites?

	I can't speak for lists.apple.com or freebsd.org, but on 
python.org it was a combination of the people who were going to be 
managing the system feeling more comfortable with postfix than 
anything else, plus our view that postfix would probably outperform 
anything else we could throw at it.

	The previous mail server for python.org used Exim, and it was not 
well-maintained.  Only one person really understood much of anything 
about it, and he wasn't around very much.  The machine itself was 
also pretty old and slow, and not well-maintained.  The new machine 
is much faster and better maintained, but I don't know how we would 
have done if we would have used Exim on the same box.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

   SAGE member since 1995.  See <http://www.sage.org/> for more info.