[Mailman-Developers] Re: mm-handler

[David Champion]

--9jxsPFA5p3P2qPhR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On 2001.07.26, in <20010726004710.B9012@smack.uchicago.edu>,
	"David Champion" <dgc@uchicago.edu> wrote:
> 
> Yes, the instructions are a bit out of date.
> 
> You can find an only-marginally-outdated version of mm-handler in the
> mailman-developers archive; the full directions on how to make it go are
> what's mainly missing here.
> 
> I'll try to put those together RSN.

Finally, here's a draft that I think should be fairly complete. This
copy of mm-handler is slightly different; it's a little less snooty and
more easily configured, and its date treatment (for generating envelope
From_ lines) is 2822-compliant instead of taking a fair shot in the Unix
tradition.

Also attached: mm-handler.readme, a sample virtusertable for RFC 2142
address exceptions, and a toy sendmail.mc file that really works.

Changes are mostly welcome, but I'd like to know just so that I can keep
my own CVS in sync with Mailman's for now. Barry, you should check step
0xA in mm-handler.readme. You need to make some changes there, I think.
:)

I'd appreciate someone's stepping through this like a newbie and noting
problems.

-- 
 -D.	dgc@uchicago.edu	NSIT	University of Chicago

--9jxsPFA5p3P2qPhR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="mm-handler.readme"

What?
-----
Mm-handler is a mail delivery agent (MDA) -- a "mailer", in Sendmail
lingo. Its function is to assume authority for messages destined to
Mailman lists, so that they're off sendmail's hands, and you (the site
administrator) don't need to maintain databases of aliases and such. It
takes a small bit of work to set up, but once that's done, you'll never
need to mess with aliases for Mailman's sake again.

When?
-----
The only further catch is that mm-handler is only really useful when
it mostly "owns" its mail domain (the hostname part after an e-mail
address's "@" symbol) -- when you can dedicate the mail domain to
Mailman. If you have a limited set of exceptions -- a few users, for
example -- you can still use it, but for sites with a dynamic or even
mix of users (or forwarders) and lists, it might not gain you much.

How do you know whether mm-handler is appropriate for you? Let's look
at some examples. If you're running lists off your primary DNS domain
name, you probably have a mix of lists and users in your namespace. Take
python.org, for example: it hosts Mailman lists, and it hosts users'
personal accounts. There aren't a whole bunch of either, but the ratio
is probably fairly near 1:1. Mm-handler is not very useful here, because
there's no simple way to separate user addresses from list addresses --
not to mention that mm-handler is written in perl, so that's just bad
form.

This begs two different, complementary situations. A hypothetical
domain, incidents.int, is used mostly for mailing lists. It's a
front-end site, and not a general user mail service. There might be
a couple of user addresses -- system administrators and such -- but
these are few in number and easily adjusted manually by the site
administrator. The 250 mailing lists at the site are much more dynamic,
and a much bigger pain to keep track of by editing an alias file. This
site can easily benefit from mm-handler.

Inversely, mail.aero, another hypothetical domain, provides POP mail
service to employees of the aerospace industry. Its addresses are
almost entirely users, although it maintains a few mailing lists for
convenience. This site has nothing to gain from mm-handler. It's much
easier to maintain an alias file of 10 lists than to dedicate the domain
to Mailman, and put all 10,000 aerospace workers in a user table.

Those a are the trickier cases. The case where mm-handler really works
best is when you can dedicate a single hostname under your DNS domain
to mailing lists, and host no user accounts there. At the University of
Chicago, we do this with listhost.uchicago.edu. SourceForge does this,
too, although I don't believe they use Sendmail.

If your site is like that, you should read on.

How?
----
Set-up isn't all that complicated. I've included a file here called
"mailman.mc". This is the m4 file that I use on my list server, and you
can likely use it with few changes at your site. It's well-annotated;
the rationale for each parameter (or set of parameters) is provided in
m4 (ahem) comments.

So, the simple steps are as follows:

1. Copy mailman.mc, and make any changes you need at your site. You
   DEFINITELY need some changes. There are hostnames in there that you
   need to adjust, and chances are that you'll need to change some other
   parameters (like the host OS), too. [1]

2. Install mm-handler. Because my server's sendmail-related files live
   in /etc/mail, I keep mm-handler there, too. YMMV.

3. Edit mm-handler, and make any changes you need at your site. You
   probably want to change $MMWRAPPER and $MMLISTDIR at line 14, and you
   *might* want to take a look at the helpful boilerplate text beginning
   at line 64. (This text is sent whenever someone tries to send mail to
   a nonexistent list address on your mail domain.)

4. You should set up a virtusertable. (See mailman.mc for an
   explanation.) There's an example of a good, minimal virtusertable
   in this distribution. The virtusertable begins as a text file named
   "virtusertable", stored in the same directory as all the other
   Sendmail files, but it's converted to a map file for Sendmail's use.
   Install the virtusertable, and (re)make the map file. [2]

5. You absolutely must have a mailertable, or all of this goes nowhere.
   Like virtusertable, the mailertable is a map that begins as text and
   gets converted. It's named "mailertable", and it's probably pretty
   simple. Mine looks like this:

	listtest.uchicago.edu	mailman:listtest.uchicago.edu

   This says: assign all incoming mail (that was not intercepted by the
   virtusertable) and that is in the listtest.uchicago.edu domain to the
   "mailman" mailer, and tell the "mailman" mailer that the hostname
   we're using is "listtest.uchicago.edu". You can support multiple
   virtual hosts using mm-handler just by placing corresponding lines in
   mailertable.

   Be sure to make this map, too!

6. The mailer definition (see the end of mailman.mc, or your own .mc
   file) for mm-handler sets the user/group that mm-handler will run
   under. (I use mailman:other.) Be sure that mm-handler is executable
   by this user or group. You almost certainly need the user to be the
   same as the Mailman user, and this user is almost always called
   "mailman", so you probably shouldn't change the defaults.

7. Generate your new sendmail.cf file. See the sendmail documentation if
   you're not familiar with this procedure. [1]

8. Stop sendmail on your list server, if you haven't already. Install
   the new sendmail.cf file wherever your sendmail.cf file belongs.
   (This depends on how sendmail was compiled, but most systems support
   using /etc/sendmail.cf.)

9. Cross your fingers and restart sendmail.

A. Barry warns that Mailman now needs you to modify your
   Mailman/mm_cfg.py file, adding this line:

	MTA = None

   But this isn't in my latest installation, and I don't remember what
   it's for, so I'll leave this to Barry to clarify. :)

That's it! With any luck, you're fully functional.

--
[1] The .mc file is the standard, supported way of configuring sendmail.
    I'm not going to get into this here, and I'm not going to tell
    you how to write raw sendmail.cf stuff, because if you need to do
    it this way then you need something more comprehensive than I can
    provide. If you need help with the .mc -> .cf process, I recommend
    these links:

	http://www.sendmail.org/~ca/email/setup1.html
	http://www.sendmail.org/~ca/email/doc8.9/README.cf.html
	http://www.hserus.net/sendmail.html

[2] This is often done with something like "makemap hash
    /etc/virtusertable </etc/virtusertable", but it could be different
    on your server. Consult the sendmail documentation if you do not
    know.

--
$Id: mm-handler.readme,v 1.2 2001/08/14 01:06:41 dgc Exp $


--9jxsPFA5p3P2qPhR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=mm-handler

#!/opt/bin/perl
##
## Sendmail mailer for Mailman
##

## Simulates these aliases:
## testlist:                "|/opt/pkgs/mailman/mail/wrapper post testlist"
## testlist-admin:          "|/opt/pkgs/mailman/mail/wrapper mailowner testlist"
## testlist-request:        "|/opt/pkgs/mailman/mail/wrapper mailcmd testlist"
## owner-testlist:          testlist-admin
## testlist-owner:          testlist-admin

## Some assembly required.
$MMWRAPPER = "/opt/pkgs/mailman/mail/wrapper";
$MMLISTDIR = "/var/mailman/lists";
$SENDMAIL = "/usr/lib/sendmail -oem -oi";
$VERSION = '$Id: mm-handler,v 1.10 2001/08/14 00:43:08 dgc Exp $';

## Comment this if you offer local user addresses.
$NOUSERS = "\nPersonal e-mail addresses are not offered by this server.";

# uncomment for debugging....
#$DEBUG = 1;

use FileHandle;
use Sys::Hostname;
use Socket;

($VERS_STR = $VERSION) =~ s/^\$\S+\s+(\S+),v\s+(\S+\s+\S+\s+\S+).*/\1 \2/;

$BOUNDARY = sprintf("%08x-%d", time, time % $$);

## Informative, non-standard rejection letter
sub mail_error {
	my ($in, $to, $list, $server, $reason) = @_;
	my $sendmail;

	if ($server && $server ne "") {
		$servname = $server;
	} else {
		$servname = "This server";
		$server = &get_ip_addr;
	}

	#$sendmail = new FileHandle ">/tmp/mm-$$";
	$sendmail = new FileHandle "|$SENDMAIL $to";
	if (!defined($sendmail)) {
		print STDERR "$0: cannot exec \"$SENDMAIL\"\n";
		exit (-1);
	}

	$sendmail->print ("From: MAILER-DAEMON\@$server
To: $to
Subject: Returned mail: List unknown
Mime-Version: 1.0
Content-type: multipart/mixed; boundary=\"$BOUNDARY\"
Content-Disposition: inline

--$BOUNDARY
Content-Type: text/plain; charset=us-ascii
Content-Description: Error processing your mail
Content-Disposition: inline

Your mail for $list could not be sent:
	$reason

For a list of publicly-advertised mailing lists hosted on this server,
visit this URL:
	http://$server/

If this does not resolve your problem, you may write to:
	postmaster\@$server
or
	mailman-owner\@$server


$servname delivers e-mail to registered mailing lists
and to the administrative addresses defined and required by IETF
Request for Comments (RFC) 2142 [1].
$NOUSERS

The Internet Engineering Task Force [2] (IETF) oversees the development
of open standards for the Internet community, including the protocols
and formats employed by Internet mail systems.

For your convenience, your original mail is attached.


[1] Crocker, D. \"Mailbox Names for Common Services, Roles and
    Functions\".  http://www.ietf.org/rfc/rfc2142.txt

[2] http://www.ietf.org/

--$BOUNDARY
Content-Type: message/rfc822
Content-Description: Your undelivered mail
Content-Disposition: attachment

");

	while ($_ = <$in>) {
		$sendmail->print ($_);
	}

	$sendmail->print ("\n");
	$sendmail->print ("--$BOUNDARY--\n");

	close($sendmail);
}

## Get my IP address, in case my sendmail doesn't tell me my name.
sub get_ip_addr {
	my $host = hostname;
	my $ip = gethostbyname($host);
	return inet_ntoa($ip);
}

## Split an address into its base list name and the appropriate command
## for the relevant function.
sub split_addr {
	my ($addr) = @_;
	my ($list, $cmd);

	if ($addr =~ /(.*)-admin$/
	    || $addr =~ /(.*)-owner$/
	    || $addr =~ /^owner-(.*)$/) {
		$list = $1;
		$cmd = "mailowner";
	} elsif ($addr =~ /(.*)-request$/) {
		$list = $1;
		$cmd = "mailcmd";
	} else {
		$list = $addr;
		$cmd = "post";
	}

	return ($list, $cmd);
}

## The time, formatted as for an mbox's "From_" line.
sub mboxdate {
	my ($time) = @_;
	my @days = qw(Sun Mon Tue Wed Thu Fri Sat);
	my @months = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
	my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst) =
		localtime($time);

	## Two-digit year handling complies with RFC 2822 (section 4.3),
	## with the addition that three-digit years are accommodated.
	if ($year < 50) {
		$year += 2000;
	} elsif ($year < 1900) {
		$year += 1900;
	}

	return sprintf ("%s %s %2d %02d:%02d:%02d %d",
		$days[$wday], $months[$mon], $mday,
		$hour, $min, $sec, $year);
}

BEGIN: {
	$sender = undef;
	$server = undef;
	@to = ();
	while ($#ARGV >= 0) {
		if ($ARGV[0] eq "-r") {
			$sender = $ARGV[1];
			shift @ARGV;
		} elsif (!defined($server)) {
			$server = $ARGV[0];
		} else {
			push(@to, $ARGV[0]);
		}
		shift @ARGV;
	}

	if ($DEBUG) {
		$to = join(',', @to);
		print STDERR "to: $to\n";
		print STDERR "sender: $sender\n";
		print STDERR "server: $server\n";
		exit(-1);
	}

ADDR:	for $addr (@to) {
		$prev = undef;
		$list = $addr;

		$cmd= "post";
		if (! -f "$MMLISTDIR/$list/config.db") {
			($list, $cmd) = &split_addr($list);
			if (! -f "$MMLISTDIR/$list/config.db") {
				$was_to = $addr;
				$was_to .= "\@$server" if ("$server" ne "");
				mail_error(\*STDIN, $sender, $was_to, $server,
					"no list named \"$list\" is known by $server");
				next ADDR;
			}
		}

		$wrapper = new FileHandle "|$MMWRAPPER $cmd $list";
		if (!defined($wrapper)) {
			## Defer?
			print STDERR "$0: cannot exec ",
				"\"$MMWRAPPER $cmd $list\": deferring\n";
			exit (-1);
		}

		# Don't need these without the "n" flag on the mailer def....
		#$date = &mboxdate(time);
		#$wrapper->print ("From $sender  $date\n");

		# ...because we use these instead.
		$from_ = <STDIN>;
		$wrapper->print ($from_);

		$wrapper->print ("X-Mailman-Handler: $VERSION\n");
		while (<STDIN>) {
			$wrapper->print ($_);
		}
		close($wrapper);
	}
}

--9jxsPFA5p3P2qPhR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="mailman.mc"

dnl
dnl *** EXAMPLE *** sendmail.mc file for a Mailman list server using
dnl mm-handler to deal with list operations (in place of aliases).
dnl This is what I actually use on my site.
dnl
dnl $Id: mailman.mc,v 1.2 2001/08/14 00:50:34 dgc Exp $
dnl


dnl
dnl First you need to define your general characteristics.  You
dnl should know what these settings should be at your site -- I
dnl only know what they should be at mine.
dnl
OSTYPE(solaris2)dnl
DOMAIN(generic)dnl

dnl
dnl You can keep the old alias files for back-compatibility, but it's
dnl probably better not to as this can become a point of confusion
dnl later.
dnl
define(`ALIAS_FILE', `/etc/mail/aliases,/etc/mail/lists')

dnl
dnl I use procmail for local delivery, because it's smart to have a
dnl local delivery mailer, even if you don't (ordinarily) do any local
dnl delivery. The Solaris local delivery mailer is part of its sendmail
dnl package. I pkgrmed the sendmail packages so that system upgrades
dnl don't kill my sendmail.com sendmail, so mail.local is unavailable,
dnl so I throw procmail in here even though it never gets used.
dnl
define(`PROCMAIL_MAILER_PATH', `/opt/bin/procmail')
FEATURE(`local_procmail')
MAILER(`local')

dnl
dnl Miscellaneous tuning.  Not relevant to Mailman.
dnl
define(`confCONNECTION_RATE_THROTTLE', 5)
define(`confMAX_MESSAGE_SIZE', `5000000')
define(`confNO_RCPT_ACTION', `add-to-undisclosed')
define(`confME_TOO', `True')
define(`confDOUBLE_BOUNCE_ADDRESS', `mailer-daemon')

dnl
dnl Privacy options.  Also not relevant.
dnl
define(`confPRIVACY_FLAGS', `authwarnings,needvrfyhelo,noexpn,noreceipts,restrictmailq')


dnl
dnl Mm-handler works by mailertabling all addresses on your list
dnl server hostname(s) through the mm-handler mailer. Mailertable
dnl maps mail domains to mailer types. I want a mailertable to map
dnl listtest.uchicago.edu to the mm-handler mailer, but we need to
dnl specifically request this functionality in the .mc file.
dnl
FEATURE(`mailertable', `hash -o /etc/mail/mailertable')

dnl
dnl This leads to an immediate and important side-effect: "local"
dnl addresses, and notably RFC-specified addresses such as postmaster,
dnl are assumed by sendmail to be lists! Since aliases are not processed
dnl for domaintabled domains, we must use a virtusertable to reroute
dnl such addresses.
dnl
FEATURE(`virtusertable', `hash -o /etc/mail/virtusertable')

dnl
dnl By default, sendmail applies virtusertable mapping, if at all, for
dnl all interfaces for which it accepts mail -- i.e., all domains in
dnl $=w. Mm-handler relies on your having a single domain (hostname)
dnl that serves only lists, with no users. To avoid potential namespace
dnl conflicts, you need not to have this list domain included in $=w.
dnl As a result, virtuser mapping does not apply for the Mailman
dnl list domain. However, you can pre-empt this rule by defining
dnl $={VirtHost}: if there are domains in this class, they will be
dnl mapped before $=w is mapped.
dnl
dnl VIRTUSER_DOMAIN() defines this class.
dnl
VIRTUSER_DOMAIN(`nospam.uchicago.edu listtest.uchicago.edu listhost.uchicago.edu')

dnl
dnl On a related point: by default, Sendmail probes for open IP
dnl interfaces, and adds their hostnames to $=w. Although Sendmail does
dnl virtusertable mapping for members of $=w, it doesn't do mailertable
dnl mapping for them, because they're considered "local". This tells
dnl Sendmail not to probe interfaces for local hosts, and it's critical
dnl if your Mailman domain is actually an IP address (with an A record,
dnl not just CNAME or MX) on your server.
dnl
define(`confDONT_PROBE_INTERFACES', `True')


dnl
dnl Even though my actual hostname is foobar, tell the world that I'm
dnl listtest.uchicago.edu.
dnl
FEATURE(`limited_masquerade')
MASQUERADE_AS(`listtest.uchicago.edu')


dnl
dnl Access control is a useful feature for blocking abusers and relays
dnl and such.
dnl
FEATURE(`access_db')


dnl
dnl This allows you to block access for individual recipents through
dnl the same access database as is used for blocking sender hosts and
dnl addresses.
dnl
FEATURE(`blacklist_recipients')


dnl
dnl Other local mailers...
dnl
MAILER(`smtp')
MAILER(`procmail')


dnl
dnl Our Mailman-specific local mailer.
dnl
MAILER_DEFINITIONS
####################################
###   New Mailer specifications  ###
####################################

## Special flags! See
##	http://www.sendmail.org/~ca/email/doc8.10/op-sh-5.html#sh-5.4
## Note especially the absence of the "m" and "n" flags. THIS IS
## IMPORTANT: mm-handler assumes this behavior to avoid having to know
## too much about address parsing and other RFC-2822 mail details.

Mmailman,	P=/etc/mail/mm-handler, F=rDFMhlqSu, U=mailman:other,
		S=EnvFromL, R=EnvToL/HdrToL,
		A=mm-handler $h $u


--9jxsPFA5p3P2qPhR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=virtusertable

##
## Example virtusertable for use with a Mailman site running mm-handler.
##
## $Id: virtusertable,v 1.1 2001/08/14 00:43:08 dgc Exp $
##

##
## My server's hostname is nospam, but we don't honor that as a
## Mailman mail domain. Anything @nospam.uchicago.edu should be
## forwarded to our master Mailman admin address.
##
@nospam.uchicago.edu			mailman-owner@midway.uchicago.edu

##
## Redirect mail to the standard Mailman admin addresses to the
## master admin address. (Midway.uchicago.edu is our site's central
## mail-routing server, and it carries aliases for maintenance groups.
## Not a good plan to entrust Mailman maintenance mail to Mailman.)
##
mailman@listhost.uchicago.edu		mailman-owner@midway.uchicago.edu
mailman-owner@listhost.uchicago.edu	mailman-owner@midway.uchicago.edu

##
## These addresses are required or recommended either by convention
## or by RFC 2142, "Mailbox Names for Common Services, Roles and
## Functions". Honor them.
##
MAILER-DAEMON@listhost.uchicago.edu	mailman-owner@midway.uchicago.edu
postmaster@listhost.uchicago.edu	mailman-owner@midway.uchicago.edu
webmaster@listhost.uchicago.edu		mailman-owner@midway.uchicago.edu
abuse@listhost.uchicago.edu		sun-admin@midway.uchicago.edu
root@listhost.uchicago.edu		sun-admin@midway.uchicago.edu

##
## If I had a need, I could put user accounts in here, too.
##
dgc@listhost.uchicago.edu		dgc@where.my.mail.really.goes

--9jxsPFA5p3P2qPhR--