[Mailman-Developers] MimeFilter integration with Mailman

[J C Lawrence]

<<Reply-To set to mailman-users@python.org>>

A while back I noticed that while we'd been happily recommending
MimeFilter for stripping unwanted MIME parts from messages before they
got to mailman, nobody had bothered to say how to run it beyond "chuck
it in the alias file".  

  Problem: those instructions (generally) won't work.

  Reason: Mimefilter tries to create files and a temp directory in its
  CWD, and at least under postfix that means /var/spool/postfix which
  here is root.root/0755 (and I'm not about to change that).

So I spent some time this morning and threw something together a little
slicker.  Its essentially a wrapper around Mailman's wrapper.  What I
got seems to work.  I don't and won't claim beauty.

--<cut>--
#!/bin/bash

# Syntax:
#
#   mimestrip <action> <target>
#
#     Both arguments will be passed to /var/lib/mailman/mail/wrapper 
#     in the same order, without change.
#
#     All other arguments will be gleefully discarded.
#
# Author: J C Lawrence <claw@kanga.nu>
#

ACTION=$1
TARGET=$2
export PATH=/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/sbin:/usr/sbin

# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# Config (Edit this bit to suit your setup!)
# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

# Safe working directory for mimefilter.  We need to do this as 
# postfix by default will execute this script in /var/spool/postfix and 
# mimefilter will try and create temp directories and files under there.

# Note: Make sure that the value of tmpdir is writeable by the UID or GID 
# that this script will execute under.

tmpdir="/tmp/mimestrip.${$}"

# Do you want to save an mbox of the messages before filtering 
# in $savefile_pre?   (0/1)

save_pre=1

# Do you want to save an mbox of the messages after filtering 
# in $savefile_post?  (0/1)

save_post=1

# File to save pre-filtered messages in

savefile_pre="/tmp/mimestrip.prefilter"

# File to same post filtered messages in

savefile_post="/tmp/mimestrip.postfilter"

# Send a copy of every processed message to the list maintainer?  (Y/N)

copy_maintainer="y"

# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# MimeFilter setup (Don't edit from here down)
# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

# The name of the mailing list this message is intended for.  Used
# as the return address of the warning issued to the orginal author
# if the message is not already clean.

export list=${USER}

# The address of the mailing list this message is intended for.
# Used in the X-Loop field of the warning issued to the original
# author if the message is not already clean.

export listaddr=${USER}@${DOMAIN}

# The administrative (owner) address of the mailing list this
# message is inteded for.  Used in the return address of the warning
# issued to the original author if the message is not already clean.

export listreq=${USER}-admin@${DOMAIN}

# The email address of the maintainer of the mailing list this
# message is inteded for.  If it is defined, it is used to send the
# maintainer original carbon copies of messages that have been
# modified by this filter -- if filter_mime_cc_maintainer is
# affermative, of course.

export maintainer=${USER}-owner@${DOMAIN}

# A boolean flag: if affermative (i.e., if it matches the /y/i Perl
# regular expression), the mimefilter script will send carbon copies
# of every cleaned (modified) message to the maintainer of the
# mailing list the message is intended for.

export filter_mime_cc_maintainer=${copy_maintainer}

# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# Start working
# -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

# Build a safe working directory for mimefilter.  We need to do this as 
# postfix by default will execute this script in /var/spool/postfix and 
# mimefilter will try and create temp directories and files under there,
# which will fail on permissions.

# Note: Make sure that the value of tmpdir is writeable by the UID or GID 
# that this script will execute under.

rm -rf ${tmpdir} 2> /dev/null
mkdir ${tmpdir}
cd ${tmpdir}

# Working files (over-engineered and IO heavy, but useful during debugging 
# and tuning of the filter setup)

tmpfile="mimestrip.$$"
holdfile="mimehold.$$"

# DEBUG

#pwd > /tmp/mimestrip.pwd
#set > /tmp/mimestrip.env

# Get the inbound message from stdin and run it thru mimefilter with
# copies saved for before and after.  The mimestrip.err file should 
# always be empty.  I found it useful for debugging.  Nuke freely.

# Note: If there's an "X-MimeStrip: No" header (actually anywhere in the
# message), then don't run this message thru mimefilter but pass it 
# straight on to the mailman wrapper.

cat > ${holdfile}

grep -i "^X-MimeStrip:[ \t]*No" ${holdfile}
if [ $? ]
then
  cat ${holdfile} | /usr/bin/mimefilter > ${tmpfile} 2> /tmp/mimestrip.err

  # If there's nothing left of the message (less than 5 lines), don't 
  # bother sending it forward to Mailman.

  count=`cat ${tmpfile} | wc -l`
  if [ $count -gt 5 ]
  then
    cat ${tmpfile} | /var/lib/mailman/mail/wrapper ${ACTION} ${TARGET}
  fi
else

  # We found a header so send it thru unfiltered.

  cat ${tmpfile} | /var/lib/mailman/mail/wrapper ${ACTION} ${TARGET}
fi

# Save copies of the before and after messages off to a temp file for 
# analysis/debugging

if [ $save_pre -eq 1 ]
then
  cat ${holdfile} >> /tmp/mimestrip.prefilter
  echo >>/tmp/mimestrip.prefilter
fi
if [ $save_post -eq 1 ]
then
  cat ${tmpfile} >> /tmp/mimestrip.postfilter
  echo >> /tmp/mimestrip.postfilter
fi

# Clean up

rm -rf ${tmpdir}
exit 0
--<cut>--

Notes:

  Use in your Mailman alias file as so (for a list called "test"):

    test:                    "|/some/path/here/mimestrip post test"

  More simply just replace the normal /var/lib/mailman/mail/wrapper"
  with the call to mimestrip.

  The script is Postfix specific in that it expects the $USER and
  $DOMAIN in its environment to be respectively set to the LHS and RHS
  of the address the message was sent to.  Hack freely for your MTA of
  choice if it does something different.

  Why not use demime which does all sorts of cool things like Yahoo
  signature removal and getting rid of unnecessary quoted-printable?
  demime strips all MIME parts, which is fine if that's what you want.
  However there are certain MIME parts that I specifically want to let
  thru to my lists (eg message/rfc822), and I rather like some of
  mimefilter's other features thta demime doesn't share like emailing
  the original poster with a warning that parts were removed from their
  message.

    Yeah, those extra messages are annoying to the posters.  They're
    intended to be and I like that.  I don't want posters sending
    unwanted MIME parts to my lists.  

    Note: You can turn off that feature in Mimefilter's config.

  I also wanted a way to send MIME messages thru to my lists without
  doing any MIME stripping.  A bypass if you would.  Typically I'd use
  this for valid/valuable list messages which contain useful MIME parts
  (eg an attached image such as a graph diagramming the message content)
  that would otherwise be stripped my the mimefilter.  In the end I did
  this via a custom X-header: X-MimeStrip: No.

    Strong suggestion: Configure your lists to hold messages with that
    header for moderator approval.  This will allow you to catch abuse
    by too-clever posters.

  Mailman v2.1 largely removes need for this mimestrip filter.  If all
  you want is mimestripping then Mailman 2.1 should be more than enough.
  However I want more than that.  I want mimefilter's annoying messages
  to posters of unwanted MIME, I want the bypass, I want the ability to
  save the before and after filtering messages etc, and I really want
  the ability to ad-hoc extend the filter with other little toys and
  trinkets as I go along (eg a quoted-printable flattener will likely be
  next for me here).

  Adding in something like SpamAssassin into the mail processing pipe
  with a custom handling rule (eg silently discard messages with high
  enough scores before passing them thru mimefilter) would be trivially
  easy and is left as an exercise for the reader.

  This is barely if at all tested.  Caveat Emptor and all that rot.

  Yeah, I should put this in the FAQ and I hopefully will after I've run
  it thru a few more exercises here.

Enjoy.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
claw@kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.