Graham's spam filter

Karl Vogel vogelke at dnaco.net
Tue Aug 27 22:10:14 CEST 2002


>> On Mon, 26 Aug 2002 13:32:27 GMT, 
>> "Edward K. Ream" <edream at tds.net> said:

E> I am an enthusiastic user of ChoiceMail,
E> http://www.digiportal.com/choicemail.html

E> It has _completely_ eliminated all spam to me, using a very clever
E> trick.  Anyone wanting to contact me who is not already on a "whitelist"
E> gets sent a return email asking them to fill out a short form at the
E> ChoiceMail site.

   The TMDA package does this as well.  I've included a short blurb.

-- 
Karl Vogel               ASC/YCOA, Wright-Patterson AFB, OH 45433
vogelke at dnaco.net                   http://www.dnaco.net/~vogelke

I will not celebrate meaningless milestones.
                                    --written on blackboard by Bart Simpson

---------------------------------------------------------------------------
http://software.libertine.org/tmda/
Tagged Message Delivery Agent (TMDA)
Jason R. Mastaler

Background & Overview
 
  TMDA is an OSI certified software application designed to significantly
  reduce the amount of SPAM/UCE (junk-mail) you receive.  TMDA combines a
  "whitelist" (for known/trusted senders), a "blacklist" (for undesired
  senders), and a cryptographically enhanced confirmation system (for
  unknown, but legitimate senders).

  TMDA was originally based upon a Perl script written by Thomas
  Erskine called Tagged Message Sender (TMS), but has since evolved
  into a significantly larger and more featureful application.

  TMDA (and its author) hold the following assumptions about the current
  Internet infrastructure:

  1.  You cannot keep your email address secret from spammers.

  2.  Content-based filters can't distinguish spam from legitimate mail
      with sufficient accuracy.

  3.  To maintain economies of scale, bulk-mailing is generally:
      An impersonal process where the recipient is not distinguished
      A one-way communication channel (from spammer to victim)

  4.  Spam will not cease until it becomes more expensive for spammers to
      operate, for which currently no infrastructure exists.

TMDA's Whitelist-based Strategy
``Deny everything that is not explicitly allowed''

  With TMDA, unrestricted access to your mailbox can no longer be
  assumed, a premise which spammers rely heavily upon.

  The way TMDA thwarts incoming junk-mail is simple yet very effective.
  You maintain a "whitelist" of trusted contacts which are allowed
  directly into your mailbox.  Messages from unknown senders are held
  in a pending queue until they respond to a confirmation request sent
  by TMDA.  Once they respond to the confirmation, their original message
  is deemed legitimate and is delivered to you.  Updating your whitelist
  insures they won't have to confirm future messages.  To see what
  the confirmation process looks like, try sending me a test message.
  (NOTE: Confirmed test messages are automatically discarded)

  This methodology has the advantage of being very selective about
  what it allows in, while at the same time permitting legitimate,
  but previously unknown senders to reach you.

Traditional Blacklist-based Strategy
``Allow everything that is not explicitly denied''

  Traditional anti-spam technical countermeasures are based upon
  maintaining a "blacklist" containing e-mail addresses, domains,
  and/or network subnets of known junk-mailers.  Incoming messages are
  then compared against this list and dropped if there is a match.

  The problem with this approach is that spammer's intrusion techniques
  are evolving as fast as your prevention techniques are, so the
  battle is never ending.  Maintaining the blacklist is often just as
  timeconsuming as pressing the "Delete" key on the easily recognized
  junk messages.  If wasted time is your biggest complaint with junk
  e-mail, you can see why this traditional methodology is flawed.
  If you really want effective spam control, you need something that
  doesn't rely on heuristics that spammers can work around.

Feature Highlights

  + Free software.

  + Written entirely in Python: an interpreted, interactive,
    object-oriented programming language.

  + Active and open development.

  + Integrated "blacklist" and "whitelist" functionality alleviating
    the need for a third-party delivery agent such as procmail or maildrop.

  + Supports MUA-independent, site-wide installations using qmail-relay
    rewriting, including non-qmail clients such as those running MS Windows.

  + Package installation available for Linux and FreeBSD.

  + Supports qmail virtualdomains including vpopmail and vmailmgr.

  + Supports matching patterns using Unix shell-style wildcard characters.

  + Optional delivery logging.

  + Easy customization of confirmation messages through templates.

  + Uses HMAC (Hashed Message Authentication Codes) for "cookie"
    generation and verification.

Results & Testimonials

  Results thus far have been extremely favorable.  Before TMDA my
  mailbox was getting bombarded by 700+ pieces of junk mail per month
  as the result of my e-mail address being heavily publicized on USENET
  and various Internet mailing lists.  Now I almost never see a piece
  of SPAM (perhaps 1-3 per month).

  User Testimonials

    "It's great.  I've received almost no spam since implementing it."
    Dave Sill on the qmail mailing list

    "Blocks spam better than any RBL ever did."  Adam McKenna on the
    qmail mailing list

    "Tagged Message Delivery Agent is a great way of blocking spam.
    Go check it out."  Guido van Driel on the qmail mailing list

    "http://tmda.sourceforge.net/ is the way to go.  Excellent,
    excellent work by Jason!"  Robin S.  Socha on gnu.emacs.gnus

TMDA In Use

  The following sites use TMDA as part of a service offering.  If you'd
  like to be included in this list, please let me know.

    + myprivacy.ca
    + my.gnus.org
    + mail.socha.net

Requirements & Downloads

  If you decide to give TMDA a whirl, make sure you have the necessary
  prerequisite software installed first.

  + The qmail mail transfer agent installed on your mail server.
    TMDA will not work with other MTAs such as Sendmail, Exim,
    and Postfix.  See the FAQ for why.

  + The Python programming language version 1.5.2 or higher.
    Many UNIX/ Linux distributions now come with Python pre-installed.
    To check whether it is and what version you have installed,
    try typing python at your shell's command line:

      % python
      Python 2.1.1 (#1, Aug 30 2001, 03:48:58) [C] on osf1V5
      Type "copyright", "credits" or "license" for more information.
      >>> ^D

  + The TMDA distribution itself.

Installation & Setup

  Installation from gzipped tarball

  Choose an installation location which can either be an individual
  area (such as within your home directory), or a shared area (such as
  /usr/local) for system-wide use, and change to that directory.

  1.  Unpack the TMDA distribution.

        # gunzip -c tmda-0.01.tgz | tar xvf -

  2.  If this is a system-wide install, run the compileall script to
      byte-compile all the .py files.  Byte-compilation is optional, but
      will speed TMDA's start-up time.  This will be done automatically
      by Python if you have write permissions in the installation
      directory, which is why it's not necessary for individual installs.

        # cd tmda-0.01
        # ./compileall

Server Side Configuration

  TMDA operates on the server as a qmail-style mail filter to process
  incoming mail messages (tmda-filter).

  1.  Setup your configuration file (~/.tmdarc) by reading through
      sample.tmdarc and Defaults.py.  Although greater customization
      is indeed possible, the only required entry is CRYPT_KEY.  It is
      important that your key is unique, so a utility has been provided
      to generate an unpredictable, random, and unique key for you.
      Simply run tmda-keygen and save the result to your ~/.tmdarc.

      NOTE: Make sure that you use the same .tmdarc on your mail server
      as you do on your client(s), particularly so that your CRYPT_KEY
      matches.

  2.  Populate your "whitelist" with e-mail addresses and/or wildcard
      patterns, one per line, which are considered trusted contacts and
      therefore allowed directly into your mailbox if the sender of an
      incoming message matches the list.  By default your whitelist is
      ~/.tmda /lists/whitelist.  See FAQ for details on matching syntax.

  3.  If necessary, create a ~/.tmda/lists/blacklist containing e-mail
      addresses and/or wildcard patterns of senders you want to refuse
      mail from.  The blacklist syntax rules are the same as for the
      whitelist.

  4.  Setup dot-qmail files appropriately in your home directory.

      First setup .qmail as shown below, and then decide what symbolic
      links you want to make.  The idea here is that you edit only
      the (.qmail), and then make links there from other dot-qmail
      files.  See dot-qmail(5) if you are unfamiliar with how to
      use dot-qmail files to control the delivery of mail messages.
      The simplest method involves only one link ( .qmail-default).
      In the absence of .qmail-default, you will need to make three links
      (.qmail-dated-default,.qmail-sender-default,.qmail-confirm-default)
      as shown below.

      +---------------------------------------------------------------------+
      |    dot-qmail file:     |      file contents:      |     action:     |
      |------------------------+--------------------------+-----------------|
      |                        |                          | If the message  |
      |                        |                          | is legitimate,  |
      |                        |                          | it will         |
      |                        |                          | pass-through to |
      |                        | | /path/to/tmda/bin/     | the second line |
      | .qmail                 | tmda-filter              | and get saved   |
      |                        | ./Mailbox                | to "~/Mailbox", |
      |                        |                          | otherwise it    |
      |                        |                          | goes through    |
      |                        |                          | the             |
      |                        |                          | confirmation    |
      |                        |                          | process.        |
      |------------------------+--------------------------+-----------------|
      |                        |                          | Catches jason-* |
      |                        | symbolic link            | as long as      |
      | .qmail-default         | ($ ln -s ~/.qmail ~      | another         |
      |                        | /.qmail-default)         | dot-qmail file  |
      |                        |                          | does not take   |
      |                        |                          | priority.       |
      |------------------------+--------------------------+-----------------|
      |                        |                          | Catches         |
      |                        |                          | jason-dated-*   |
      |                        | symbolic link            | and then        |
      | .qmail-dated-default   | ($ ln -s ~/.qmail ~      | forwards to     |
      |                        | /.qmail-dated-default)   | TMDA for dated  |
      |                        |                          | cookie          |
      |                        |                          | processing      |
      |------------------------+--------------------------+-----------------|
      |                        |                          | Catches         |
      |                        |                          | jason-sender-*  |
      |                        | symbolic link            | and then        |
      | .qmail-sender-default  | ($ ln -s ~/.qmail ~      | forwards to     |
      |                        | /.qmail-sender-default)  | TMDA for sender |
      |                        |                          | cookie          |
      |                        |                          | processing      |
      |------------------------+--------------------------+-----------------|
      |                        |                          | Catches         |
      |                        |                          | jason-confirm-* |
      |                        | symbolic link            | and then        |
      | .qmail-confirm-default | ($ ln -s ~/.qmail ~      | forwards to     |
      |                        | /.qmail-confirm-default) | TMDA for        |
      |                        |                          | confirmation    |
      |                        |                          | cookie          |
      |                        |                          | processing      |
      +---------------------------------------------------------------------+

Client Side Configuration

  In addition to filtering your incoming e-mail, TMDA can also work
  with your mail client to modify your outgoing address with unique,
  cryptographically enhanced (tagged) e-mail addresses.  This is done
  to compliment the filter's "whitelist" functionality.

  Dated Addresses

    TMDA can automatically tag your messages with a temporary e-mail
    address which only works for a defined time interval (e.g, 5 days).
    During this period, even those not on your whitelist will be able
    to contact you using the dated address.  This also allows you
    to participate freely in open discussion forums such as Internet
    mailing lists without fear that your address will be "harvested"
    by spammers for later use against you, and without opening up your
    whitelist more than necessary.  Here is an example dated address:

        jason-dated-989108708.a17f80 at mastaler.com

    This particular address expires on Sun, May 6 00:25:08 2001
    UTC, which is exactly 5 days after it was generated.  TMDA time
    intervals can be set in years, months, weeks, days, hours, minutes,
    and seconds.  Once a dated address expires, messages sent there must
    go through the confirmation process.  Use of strong cryptography
    insures that the timestamp can't be modified.

  Sender Addresses

    TMDA can also tag your messages with a sender address which is
    an e-mail address that only a certain sender can use.  Here is an
    example sender address:

        jason-sender-a751af at mastaler.com

    This particular sender address will only accept messages from
    president at whitehouse.gov.  Other messages must go through the
    confirmation process.

  The tmda-inject command is invoked from your client to automatically
  add the proper tag to your envelope sender address and then pipe the
  output to qmail-inject (i.e.  send your mail).

  NOTE: These instructions assume your are on a UNIX client which is
  running qmail.  If your client does not (or can not) run qmail as in
  the case of Microsoft Windows, see the FAQ for alternate instructions.

  * .tmdarc configuration:

  The tag that gets added to your address in your outgoing mail
  depends on how you have configured your .tmdarc.  Unless you tell it
  differently, TMDA will by default tag your outgoing address with a
  dated cookie.  For other options, see the "User configurable settings"
  section in Defaults.py.

  NOTE: The bare COOKIE_TYPE is especially useful when maintaining a
  "whitelist" of trusted contacts to filter your incoming mail against
  as described in the above sections.  By default, addresses matching
  your WHITELIST will receive untagged (no cookie added) messages.
  This way your trusted contacts will not even be aware of TMDA.
  Set WHITELIST_TO_BARE = 0 in your .tmdarc to reverse this behavior.

  * Mail User Agent (MUA) configuration:

  To have TMDA read your .tmdarc each time it sends outgoing mail and
  modify your sender address accordingly, you must arrange for your
  MUA to call the included sendmail compatibility wrapper.  Here is
  how to accomplish this for a few popular MUAs.  Let me know if you
  are using TMDA with a MUA not listed here.

    + For BSD Mail/mailx, add the following line to your .mailrc file:

          set sendmail="/path/to/tmda/bin/tmda-sendmail"

    + For Pine, add the following line to your .pinerc file:

          sendmail-path="/path/to/tmda/bin/tmda-sendmail"

    + For Mutt, add the following line to your .muttrc file:

          set sendmail="/path/to/tmda/bin/tmda-sendmail"

    + For VM, add the following line to your .vm file:

          (setq sendmail-program "/path/to/tmda/bin/tmda-sendmail")

    + For Gnus, add the following lines to your .gnus file:

        (setq message-send-mail-function 'message-send-mail-with-qmail)
        (setq message-qmail-inject-program "/path/to/tmda/bin/tmda-sendmail")

  * X-TMDA header:

  You can override any declarations you might have in your list files by
  adding this header when composing a message.  tmda-inject will look
  for it when determining how to tag the message.  The header will be
  removed before the message is sent.  Here is how it can be used:

    X-TMDA: bare           Sends the message to all recipients with your
                           address untagged (no cookie added)

    X-TMDA: dated          Send the message to all recipients with a dated
                           cookie added to your address

    X-TMDA: sender         Send the message to all recipients with a sender
                           cookie added to your address

    X-TMDA: exp            Send the message to all recipients from
    postmaster at xemacs.org  postmaster at xemacs.org

    X-TMDA: ext            Send the message to all recipients from
    mlist-xemacs-beta      jason-mlist-xemacs-beta at mastaler.com

  * Miscellaneous:

  Two utilities tmda-dated-address and tmda-sender-address are included
  to generate dated and sender style addresses from the command line.

  If you still read USENET (a notorious source of SPAM), you might find
  it useful to post using a dated address.  Simply have your newsreader
  call tmda-dated-address and use the result as your posting address.

  For Gnus, this could be accomplished with the following addition to
  your .gnus file:

    (defun tmda-dated-address ()
      (shell-command-to-string "/path/to/tmda/bin/tmda-dated-address"))

    (setq gnus-posting-styles
          (message-this-is-news
           (address tmda-dated-address)))

  tmda-sender-address can be used to generate subscription addresses for
  mailing lists without worry that the subscription list might get harvested
  by spammers. For example:

    $ tmda-sender-address python-list-admin at python.org | more
      jason-sender-2cd688 at mastaler.com

  (now subscribe to python-list using this sender-address, and only
  python-list-admin at python.org will be able to send mail there)

Mailing Lists

  * Several TMDA Mailing Lists are available.  tmda-list is the preferred
    forum for participating in TMDA development, reporting bugs, asking
    questions, and providing feedback.  tmda-announce is for those who
    only want to receive infrequent announcements about new releases
    and other project milestones.

Other Resources

  * TMDA's Project Management Site at Sourceforge
  * The qmail Handbook contains a section describing TMDA




More information about the Python-list mailing list