Graham's spam filter
Karl Vogel
vogelke at dnaco.net
Tue Aug 27 16:10:14 EDT 2002
>> On Mon, 26 Aug 2002 13:32:27 GMT,
>> "Edward K. Ream" <edream at tds.net> said:
E> I am an enthusiastic user of ChoiceMail,
E> http://www.digiportal.com/choicemail.html
E> It has _completely_ eliminated all spam to me, using a very clever
E> trick. Anyone wanting to contact me who is not already on a "whitelist"
E> gets sent a return email asking them to fill out a short form at the
E> ChoiceMail site.
The TMDA package does this as well. I've included a short blurb.
--
Karl Vogel ASC/YCOA, Wright-Patterson AFB, OH 45433
vogelke at dnaco.net http://www.dnaco.net/~vogelke
I will not celebrate meaningless milestones.
--written on blackboard by Bart Simpson
---------------------------------------------------------------------------
http://software.libertine.org/tmda/
Tagged Message Delivery Agent (TMDA)
Jason R. Mastaler
Background & Overview
TMDA is an OSI certified software application designed to significantly
reduce the amount of SPAM/UCE (junk-mail) you receive. TMDA combines a
"whitelist" (for known/trusted senders), a "blacklist" (for undesired
senders), and a cryptographically enhanced confirmation system (for
unknown, but legitimate senders).
TMDA was originally based upon a Perl script written by Thomas
Erskine called Tagged Message Sender (TMS), but has since evolved
into a significantly larger and more featureful application.
TMDA (and its author) hold the following assumptions about the current
Internet infrastructure:
1. You cannot keep your email address secret from spammers.
2. Content-based filters can't distinguish spam from legitimate mail
with sufficient accuracy.
3. To maintain economies of scale, bulk-mailing is generally:
An impersonal process where the recipient is not distinguished
A one-way communication channel (from spammer to victim)
4. Spam will not cease until it becomes more expensive for spammers to
operate, for which currently no infrastructure exists.
TMDA's Whitelist-based Strategy
``Deny everything that is not explicitly allowed''
With TMDA, unrestricted access to your mailbox can no longer be
assumed, a premise which spammers rely heavily upon.
The way TMDA thwarts incoming junk-mail is simple yet very effective.
You maintain a "whitelist" of trusted contacts which are allowed
directly into your mailbox. Messages from unknown senders are held
in a pending queue until they respond to a confirmation request sent
by TMDA. Once they respond to the confirmation, their original message
is deemed legitimate and is delivered to you. Updating your whitelist
insures they won't have to confirm future messages. To see what
the confirmation process looks like, try sending me a test message.
(NOTE: Confirmed test messages are automatically discarded)
This methodology has the advantage of being very selective about
what it allows in, while at the same time permitting legitimate,
but previously unknown senders to reach you.
Traditional Blacklist-based Strategy
``Allow everything that is not explicitly denied''
Traditional anti-spam technical countermeasures are based upon
maintaining a "blacklist" containing e-mail addresses, domains,
and/or network subnets of known junk-mailers. Incoming messages are
then compared against this list and dropped if there is a match.
The problem with this approach is that spammer's intrusion techniques
are evolving as fast as your prevention techniques are, so the
battle is never ending. Maintaining the blacklist is often just as
timeconsuming as pressing the "Delete" key on the easily recognized
junk messages. If wasted time is your biggest complaint with junk
e-mail, you can see why this traditional methodology is flawed.
If you really want effective spam control, you need something that
doesn't rely on heuristics that spammers can work around.
Feature Highlights
+ Free software.
+ Written entirely in Python: an interpreted, interactive,
object-oriented programming language.
+ Active and open development.
+ Integrated "blacklist" and "whitelist" functionality alleviating
the need for a third-party delivery agent such as procmail or maildrop.
+ Supports MUA-independent, site-wide installations using qmail-relay
rewriting, including non-qmail clients such as those running MS Windows.
+ Package installation available for Linux and FreeBSD.
+ Supports qmail virtualdomains including vpopmail and vmailmgr.
+ Supports matching patterns using Unix shell-style wildcard characters.
+ Optional delivery logging.
+ Easy customization of confirmation messages through templates.
+ Uses HMAC (Hashed Message Authentication Codes) for "cookie"
generation and verification.
Results & Testimonials
Results thus far have been extremely favorable. Before TMDA my
mailbox was getting bombarded by 700+ pieces of junk mail per month
as the result of my e-mail address being heavily publicized on USENET
and various Internet mailing lists. Now I almost never see a piece
of SPAM (perhaps 1-3 per month).
User Testimonials
"It's great. I've received almost no spam since implementing it."
Dave Sill on the qmail mailing list
"Blocks spam better than any RBL ever did." Adam McKenna on the
qmail mailing list
"Tagged Message Delivery Agent is a great way of blocking spam.
Go check it out." Guido van Driel on the qmail mailing list
"http://tmda.sourceforge.net/ is the way to go. Excellent,
excellent work by Jason!" Robin S. Socha on gnu.emacs.gnus
TMDA In Use
The following sites use TMDA as part of a service offering. If you'd
like to be included in this list, please let me know.
+ myprivacy.ca
+ my.gnus.org
+ mail.socha.net
Requirements & Downloads
If you decide to give TMDA a whirl, make sure you have the necessary
prerequisite software installed first.
+ The qmail mail transfer agent installed on your mail server.
TMDA will not work with other MTAs such as Sendmail, Exim,
and Postfix. See the FAQ for why.
+ The Python programming language version 1.5.2 or higher.
Many UNIX/ Linux distributions now come with Python pre-installed.
To check whether it is and what version you have installed,
try typing python at your shell's command line:
% python
Python 2.1.1 (#1, Aug 30 2001, 03:48:58) [C] on osf1V5
Type "copyright", "credits" or "license" for more information.
>>> ^D
+ The TMDA distribution itself.
Installation & Setup
Installation from gzipped tarball
Choose an installation location which can either be an individual
area (such as within your home directory), or a shared area (such as
/usr/local) for system-wide use, and change to that directory.
1. Unpack the TMDA distribution.
# gunzip -c tmda-0.01.tgz | tar xvf -
2. If this is a system-wide install, run the compileall script to
byte-compile all the .py files. Byte-compilation is optional, but
will speed TMDA's start-up time. This will be done automatically
by Python if you have write permissions in the installation
directory, which is why it's not necessary for individual installs.
# cd tmda-0.01
# ./compileall
Server Side Configuration
TMDA operates on the server as a qmail-style mail filter to process
incoming mail messages (tmda-filter).
1. Setup your configuration file (~/.tmdarc) by reading through
sample.tmdarc and Defaults.py. Although greater customization
is indeed possible, the only required entry is CRYPT_KEY. It is
important that your key is unique, so a utility has been provided
to generate an unpredictable, random, and unique key for you.
Simply run tmda-keygen and save the result to your ~/.tmdarc.
NOTE: Make sure that you use the same .tmdarc on your mail server
as you do on your client(s), particularly so that your CRYPT_KEY
matches.
2. Populate your "whitelist" with e-mail addresses and/or wildcard
patterns, one per line, which are considered trusted contacts and
therefore allowed directly into your mailbox if the sender of an
incoming message matches the list. By default your whitelist is
~/.tmda /lists/whitelist. See FAQ for details on matching syntax.
3. If necessary, create a ~/.tmda/lists/blacklist containing e-mail
addresses and/or wildcard patterns of senders you want to refuse
mail from. The blacklist syntax rules are the same as for the
whitelist.
4. Setup dot-qmail files appropriately in your home directory.
First setup .qmail as shown below, and then decide what symbolic
links you want to make. The idea here is that you edit only
the (.qmail), and then make links there from other dot-qmail
files. See dot-qmail(5) if you are unfamiliar with how to
use dot-qmail files to control the delivery of mail messages.
The simplest method involves only one link ( .qmail-default).
In the absence of .qmail-default, you will need to make three links
(.qmail-dated-default,.qmail-sender-default,.qmail-confirm-default)
as shown below.
+---------------------------------------------------------------------+
| dot-qmail file: | file contents: | action: |
|------------------------+--------------------------+-----------------|
| | | If the message |
| | | is legitimate, |
| | | it will |
| | | pass-through to |
| | | /path/to/tmda/bin/ | the second line |
| .qmail | tmda-filter | and get saved |
| | ./Mailbox | to "~/Mailbox", |
| | | otherwise it |
| | | goes through |
| | | the |
| | | confirmation |
| | | process. |
|------------------------+--------------------------+-----------------|
| | | Catches jason-* |
| | symbolic link | as long as |
| .qmail-default | ($ ln -s ~/.qmail ~ | another |
| | /.qmail-default) | dot-qmail file |
| | | does not take |
| | | priority. |
|------------------------+--------------------------+-----------------|
| | | Catches |
| | | jason-dated-* |
| | symbolic link | and then |
| .qmail-dated-default | ($ ln -s ~/.qmail ~ | forwards to |
| | /.qmail-dated-default) | TMDA for dated |
| | | cookie |
| | | processing |
|------------------------+--------------------------+-----------------|
| | | Catches |
| | | jason-sender-* |
| | symbolic link | and then |
| .qmail-sender-default | ($ ln -s ~/.qmail ~ | forwards to |
| | /.qmail-sender-default) | TMDA for sender |
| | | cookie |
| | | processing |
|------------------------+--------------------------+-----------------|
| | | Catches |
| | | jason-confirm-* |
| | symbolic link | and then |
| .qmail-confirm-default | ($ ln -s ~/.qmail ~ | forwards to |
| | /.qmail-confirm-default) | TMDA for |
| | | confirmation |
| | | cookie |
| | | processing |
+---------------------------------------------------------------------+
Client Side Configuration
In addition to filtering your incoming e-mail, TMDA can also work
with your mail client to modify your outgoing address with unique,
cryptographically enhanced (tagged) e-mail addresses. This is done
to compliment the filter's "whitelist" functionality.
Dated Addresses
TMDA can automatically tag your messages with a temporary e-mail
address which only works for a defined time interval (e.g, 5 days).
During this period, even those not on your whitelist will be able
to contact you using the dated address. This also allows you
to participate freely in open discussion forums such as Internet
mailing lists without fear that your address will be "harvested"
by spammers for later use against you, and without opening up your
whitelist more than necessary. Here is an example dated address:
jason-dated-989108708.a17f80 at mastaler.com
This particular address expires on Sun, May 6 00:25:08 2001
UTC, which is exactly 5 days after it was generated. TMDA time
intervals can be set in years, months, weeks, days, hours, minutes,
and seconds. Once a dated address expires, messages sent there must
go through the confirmation process. Use of strong cryptography
insures that the timestamp can't be modified.
Sender Addresses
TMDA can also tag your messages with a sender address which is
an e-mail address that only a certain sender can use. Here is an
example sender address:
jason-sender-a751af at mastaler.com
This particular sender address will only accept messages from
president at whitehouse.gov. Other messages must go through the
confirmation process.
The tmda-inject command is invoked from your client to automatically
add the proper tag to your envelope sender address and then pipe the
output to qmail-inject (i.e. send your mail).
NOTE: These instructions assume your are on a UNIX client which is
running qmail. If your client does not (or can not) run qmail as in
the case of Microsoft Windows, see the FAQ for alternate instructions.
* .tmdarc configuration:
The tag that gets added to your address in your outgoing mail
depends on how you have configured your .tmdarc. Unless you tell it
differently, TMDA will by default tag your outgoing address with a
dated cookie. For other options, see the "User configurable settings"
section in Defaults.py.
NOTE: The bare COOKIE_TYPE is especially useful when maintaining a
"whitelist" of trusted contacts to filter your incoming mail against
as described in the above sections. By default, addresses matching
your WHITELIST will receive untagged (no cookie added) messages.
This way your trusted contacts will not even be aware of TMDA.
Set WHITELIST_TO_BARE = 0 in your .tmdarc to reverse this behavior.
* Mail User Agent (MUA) configuration:
To have TMDA read your .tmdarc each time it sends outgoing mail and
modify your sender address accordingly, you must arrange for your
MUA to call the included sendmail compatibility wrapper. Here is
how to accomplish this for a few popular MUAs. Let me know if you
are using TMDA with a MUA not listed here.
+ For BSD Mail/mailx, add the following line to your .mailrc file:
set sendmail="/path/to/tmda/bin/tmda-sendmail"
+ For Pine, add the following line to your .pinerc file:
sendmail-path="/path/to/tmda/bin/tmda-sendmail"
+ For Mutt, add the following line to your .muttrc file:
set sendmail="/path/to/tmda/bin/tmda-sendmail"
+ For VM, add the following line to your .vm file:
(setq sendmail-program "/path/to/tmda/bin/tmda-sendmail")
+ For Gnus, add the following lines to your .gnus file:
(setq message-send-mail-function 'message-send-mail-with-qmail)
(setq message-qmail-inject-program "/path/to/tmda/bin/tmda-sendmail")
* X-TMDA header:
You can override any declarations you might have in your list files by
adding this header when composing a message. tmda-inject will look
for it when determining how to tag the message. The header will be
removed before the message is sent. Here is how it can be used:
X-TMDA: bare Sends the message to all recipients with your
address untagged (no cookie added)
X-TMDA: dated Send the message to all recipients with a dated
cookie added to your address
X-TMDA: sender Send the message to all recipients with a sender
cookie added to your address
X-TMDA: exp Send the message to all recipients from
postmaster at xemacs.org postmaster at xemacs.org
X-TMDA: ext Send the message to all recipients from
mlist-xemacs-beta jason-mlist-xemacs-beta at mastaler.com
* Miscellaneous:
Two utilities tmda-dated-address and tmda-sender-address are included
to generate dated and sender style addresses from the command line.
If you still read USENET (a notorious source of SPAM), you might find
it useful to post using a dated address. Simply have your newsreader
call tmda-dated-address and use the result as your posting address.
For Gnus, this could be accomplished with the following addition to
your .gnus file:
(defun tmda-dated-address ()
(shell-command-to-string "/path/to/tmda/bin/tmda-dated-address"))
(setq gnus-posting-styles
(message-this-is-news
(address tmda-dated-address)))
tmda-sender-address can be used to generate subscription addresses for
mailing lists without worry that the subscription list might get harvested
by spammers. For example:
$ tmda-sender-address python-list-admin at python.org | more
jason-sender-2cd688 at mastaler.com
(now subscribe to python-list using this sender-address, and only
python-list-admin at python.org will be able to send mail there)
Mailing Lists
* Several TMDA Mailing Lists are available. tmda-list is the preferred
forum for participating in TMDA development, reporting bugs, asking
questions, and providing feedback. tmda-announce is for those who
only want to receive infrequent announcements about new releases
and other project milestones.
Other Resources
* TMDA's Project Management Site at Sourceforge
* The qmail Handbook contains a section describing TMDA
More information about the Python-list
mailing list