Mailman 3 September 2002 - SpamBayes

Advantages of S?
by Gary Robinson April 26, 2003

April 26, 2003

At present, it appears that S may perform better than relatively untweaked graham (as I guess bogofilter is using), but about the same as tweaked graham as spambayes is using, but has the advantage of showing a spread of results. As I see it there are two possible advantages to the spread: 1) emails in the middle could be classified as "you should manually check these", or 2) the user could adjust for the level of sensitivity he wants (similar to what happens now in Entourage which is otherwise brain-dead as a spam filter). (1) is a attractive but one thing worries me: it may give people a false sense of security -- they may think they only have to look at the middle ones, but occasionally even extreme cases of seeming to be spam might be legit. On the other hand, on those rare occasions, the sender can always try again if he doesn't get a response! ;) Just a couple of thoughts. --Gary -- Gary Robinson CEO Transpose, LLC grobinson(a)transpose.com 207-942-3463 http://www.emergentmusic.com http://radio.weblogs.com/0101454

8 19

Central Limit Theorem??!! :)
by Gary Robinson Oct. 21, 2002

Oct. 21, 2002

Guys, I happened to have an idea while leaving McDonald's today that, after toying with it a bit, seems so clearly right, and is so exciting to me, that I just have to mention it. For theoretical reasons that you will see, and which are so straightforward there can really be little doubt about them, it should perform really, really well. Here goes: First, see this description of the Central Limit Theorem: > > The central limit theorem states that given a distribution with a mean m and > variance s^2, the sampling distribution of the mean approaches a normal > distribution with a mean (m) and a variance (s^2)/N as N, the sample size, > increases. (edited slightly from http://davidmlane.com/hyperstat/A14043.html for compatibility with this text medium.) The great thing about this is that the original variables don't have to be normally distributed. They can be ANY distribution... including the distribution we happen to have in our f(w)'s. Even if we only take the most extreme f(w)'s! This gives us a means to be totally rigorous, even based on the very application-specific, not-mathematically-rigorous data we are starting with. Say we have decided only to look at the 30 most extreme f(w)'s in every email. I am choosing 30 because it is usually enough for the central limit theorem to really kick in, but we may get better performance with a greater number of extrema. Now let's generate a population of f(w)'s as follows. We go through all the spam emails one by one, adding the 30 most extreme f(w)'s to our population (only making one addition for each unique occurrence of a word in the population). There are certainly enough data points in a training corpus of > 1,000 emails that this would give us a nice normal distribution. So, from this population, we calculate the population mean M, and the population variance S^2. Now we look at a particular email, and consider the 30 most extreme f(w)'s in that email to be a sample from the larger population (again counting each word only once). The sample mean m (calculated by simply averaging the f(w)'s) has a mean M and variance (s^2)/n. Normally n will be 30, or course, but for emails with less than 30 unique words it will obviously be less. Now calculate the z-score of m, using M as the mean and (s^2)/n as the variance. Now do the all the same things, but from the perspective of considering the ham corpus to be the population. (Obviously we would compute the ham and spam population M's and S^2's up front and only do them once.) So we have two z-scores: zs and zh for spam and ham respectively. Let R = abs(zh) - abs(zs). If R > 0, we have spam; otherwise we have ham. That is, we accept the ham/spam hypothesis that is less violated by the data. This really should work very, very well, because it incorporates much more data than something like S does (or Graham's combining mechod) and uses it in a much more meaningful way. It will work less well as the number of words in an email approach 0 because the central limit theorem becomes progressively less applicable in such cases, but that's no big surprise, is it? ;) And most such emails are probably so clearly spam or ham that this degradation will not be an issue. It is a little complicated to implement but really not that complicated. Any elementary statistics book contains straightforward arithmetic techniques for computing these numbers. Moreover, Python probably already has libraries that can compute them effortlessly. I would be extremely interested in the results of any testing on this. --Gary -- Gary Robinson CEO Transpose, LLC grobinson(a)transpose.com 207-942-3463 http://www.emergentmusic.com http://radio.weblogs.com/0101454

13 111

Re: [Spambayes] Good evening/morning/afternoon everyone
by Josiah Carlson Oct. 3, 2002

Oct. 3, 2002

Richie, > Rather than modifying the Subject line, I add an extra header. I believe > all email clients that have any filtering ability will allow you to filter > on an arbitrary header. This is also how Hammie, an alternative driver > for the spambayes code written by Neale Pickett, works (Hammie is designed > to fit into a Unix mail delivery system). The header that pop3proxy.py > adds is called X-Hammie-Disposition, so that it matches Hammie. The popfile.sourceforge.net project does a similar thing, adding an X-Text-Classification header (he is also submitting it as an RFC). I originally had that be the default behavior of PASP, but remembered that I have (in the past) had email software that doesn't allow arbitrary header matching. By inserting the Subject, I guarantee that ANY email software can filter it. > pop3proxy.py adjusts the message sizes reported by STAT and LIST to > include the added X-Hammie-Disposition header. I'm not sure this makes > any real difference, but there may be an email client out there that > malloc()s just enough space and then reads the message into it... I never thought of that. Looks like I'm going to have to do some adjustment. Cursed C. For those emails that turn out to not be spam, I am thinking that it wouldn't be a big deal to just put some blank spaces at the end of the email. I hope that no one would really notice. > I like your system of adjusting the USER to allow proxying to multiple > accounts - pop3proxy.py doesn't do this. That is actually another cue I got from popfile. To be truthful, I started working on PASP because I wanted something that worked like popfile, but was free. At the time, popfile was closed source, but has recently gone open source. > My email client (Forte Agent) times out after a couple of minutes of > inactivity. I guess others do too. This caused a problem with large > messages (eg. with attachments) - the proxy would take so long downloading > the message without sending anything to the client that the client would > time out. For this reason, I added a timeout to pop3proxy.py for long > messages - it downloads as much as it can in 30 seconds, then classifies > according to the data it's received and starts sending data back to the > client. I also hadn't thought about timeouts. I'm on 1mbit DSL and never receive emails larger than 200k, and this only occurs twice a week when I have to update my GF's webcomic (www.blue-comic.com if you are curious). The timeout idea is good, especially for anyone with a dialup, slow dsl connection, or even shared connections with a packetshaper or heavy traffic. I am going to look into adding that. > I'm told that MSN uses a proprietary POP3 authentication protocol - I > couldn't find much information about it on the web, but I think it's based > on POP3-AUTH. I think pop3proxy.py should support it, because it accepts > any command and assumes that unknown commands are not multiline. I think > your code does the same - is that right? - your 'supported' dictionary > seems to be unused. Do you know anything about this MSN system, or do you > have the means to test against it ('cos I don't...) I have no idea. I know that Hotmail requires either pay for use pop3 email, or you have to use Outlook [Express]. It wouldn't be too much work to do some network sniffing and reverse-engineer the protocol, as I've been looking to add in a hotmail/yahoomail portion in the future. I don't suppose anyone out there is willing to take YahooPOPS! and do a code translation into python... Currently the code doesn't check for supported/unknown commands, but the below modification adds it... elif self.lcmd in supported or self.lcmd == 'usent': self.client.send(data) self.client.send('\r\n') else: self.client.send('-ERR command not supported, goodbye\r\n') self.client.close_when_done() #and all the other stuff to kill the connection to the server. > A question: could you explain 'usent'? I see self.lcmd being set to > 'usent', but that seems to be the only place it gets used. Is it trying > to cope with POP3 servers that print an extra line after the response to > the USER command? Ahhh. It is mostly just a placeholder. Because I wanted the mail client to see the pop3 proxy greeting, the pop3 server greeting, and the pop3 server USER response I had to do a couple things. I also didn't want to send extra +OK messages on different lines, in case the client had problems with such things. The following is the type of thing you would capture in your client logs. +OK pop3 proxy ready USER randomuser:pop3.rs.com +OK pop3 server at pop3.rs.com +OK password required for randomuser PASS ****** +OK Contgratulations. STAT The second +OK line is what the 'usent' is for. It doesn't need it until I add in the lines earlier in this email. You don't have to deal with this because your pop3proxy KNOWS which server to connect to before the client connects. PASP doesn't know until it gets the USER user:server command. > Another question: "pop_proxy.callback(self.parent, ...)" seems like an > unusual way of writing "self.parent.callback(...)" - is this just a > stylistic difference, or is there a special reason you do it this way? I started with self.parent.callback(...), but I kept getting errors with the parent not being able to access self.server and self.client at all. pop_proxy.callback(self.parent,...) works, and I haven't bothered to see if I can change it back and still have it work. I'm lazy like that. - Josiah

5 9

Matt Sergeant: Introduction
by Matt Sergeant Oct. 3, 2002

Oct. 3, 2002

Just wanted to stick my hand up and say "Hi". I've been following this list on gmane.org for a while now (it's a mail to nntp gateway for those interested in following multiple technical mailing lists in a read-only fashion), but decided it was time to bite the bullet and subscribe, mostly because I keep itching to reply to certain posts :-) First of all, I'm a perl guy (boo, hiss). Some of you may or may have heard of me from the perl community (I won the ActiveState awards for Perl and XSLT this year, though I didn't really deserve the XSLT one <grin>). I'm also actively involved in the SpamAssassin project. By day I work as an anti-spam technologist (one of the lucky few to actually work day-in day-out to fight this nuisance) for MessageLabs. I was doing Bayesian probability techniques for spam detection before Paul Graham published his article and set the world on fire. Like you all, I discovered very quickly that it's the tokenisation techniques that are the biggest "win" when it comes down to it. To answer the curious, yes we're going to add some sort of bayesian technique to spamassassin - one of our developers has written code independantly from my stuff at work (because I'm contract bound not to give that away) that uses SpamAssassin in a "training" mode before switching on to full scoring mode. It'll basically work much like the other SA rules, where the probability gives a score. If we go with Graham it'll likely be boolean, if we go with Robinson we can give a gradual score range. But that's mostly out of my hands (due to day-job conflicts). Anyway, just wanted to say "Hi", and to let you know that I have converted the PG (final) code to Perl, and it worked well, and I've done the Robinson stuff without the central limit theorem and it didn't work quite as well, so I'm hopefully going to get CLT done this week and see how it fares. Unfortunately I find python incredibly difficult to read, so it takes me a while! Oh, and I'll also be talking at Paul Graham's spam conference about doing spam detection at the internet level (at messagelabs you point your MX to us, and we "clean" your email then forward it on), and the issues that gives rise to, such as how the probability stuff works so much better on individuals' corpora (or on a particular mailing list's corpus) than it does for hundreds of thousands of users. Have fun, Matt.

7 14

Central limit
by Rob Hooft Oct. 2, 2002

Oct. 2, 2002

While my computer is eating away cycles running timcv on my 24000 ham/ 5750 spam messages, a few remarks on the Central Limit Theorem: - The standard deviations seem "underestimated". Gary already said this can be caused by correlations between scores. Alternatively this can indicate that the data is not 1D: in more than one dimension, a higher percentage of normally distributed data lies outside of the "core regions". Anyway, something can be done about this: just calculate the RMS Z-score, and scale it to 1.0. After applying that scale the normal 68%/95% rule should apply. One would hope. - The "certainty" rule of Tim should be formalized. This is a very powerful concept not only for those people that believe in a middle ground, but also for those that want to use a pre-trained spambayes system for other corpora. Once this has crystallized a bit more we should exchange pickles and see how well we can do with each-others training data! - It should somehow be possible to classify messages into any sumber of distinct groups using this trick. A new message can get scored a Z-score to describe the likelyhood that it is part of any of the groups, if all of these numbers are large, the test message does not belong to any class. I guess, e.g. that it should not be too difficult for the bayesian algorithms used here to judge whether E-mail I receive is for "work", "private" or "spam". What would take this really to the next generation would be an algorithm that can make the classification "ab initio" as a sort of clustering algorithm: e.g. something that would start with two of the most different messages in a single corpus, and add single messages to either of the two groups until it finds a message that has 2 large Z-scores. Then it starts a third group. Just dreaming. Rob -- Rob W.W. Hooft || rob(a)hooft.net || http://www.hooft.net/people/rob/

10 22

mining dates?
by Skip Montanaro Oct. 2, 2002

Oct. 2, 2002

I notice when I arrive in the morning that a fairly large number of spam messages are waiting for me, already isolated, courtesy SpamAssassin. I thought, "hmmm... I wonder if more spam is sent overnight than during the day?", so I performed a simple experiment. I "mined" the Date: headers from my ham and spam collections and plopped the times into 6-minute buckets (local time) like so: find Data/Ham -type f \ | xargs cat \ | sed -n -e '/^From /,/^$/p' \ | egrep '^Date: ' \ | sed -e 's/.*$[0-9][0-9]:[0-9][0-9]:[0-9][0-9]$.*/\1/' \ | awk -F: '{print int(($1*60+$2)/10)}' \ | sort -n \ | uniq -c \ | awk '{print $2, $1}' > hamhour.plt and then plotted the output of both the ham and spam collections using gnuplot. The result is attached. It didn't prove my hypothesis, but may have exposed something as useful. Spam seems to be sent at a fairly constant rate throughout the day, which stands to reason, since it's probably all sent automatically. However, ham definitely seems to be sent predominantly during waking hours (doh!). I'm going to give a little date mining a try. Skip

5 9

to From_ or not to From_?
by Skip Montanaro Oct. 1, 2002

Oct. 1, 2002

I just thought of something. If I'm using the MH-style setup (one message per file in Data/{Ham,Spam}/Set* as understood by timcv.py) are my messages supposed to have From_ lines or not? Skip

6 9

Some ideas I have....
by John Draper Oct. 1, 2002

Oct. 1, 2002

HI, I want to start up another discussion about what the direction of the group is heading, as far as addressing the issues of where spam filter should take place. IE: Client side, Vs Server side. We've all been through this before, but I want a little more clarity the group is heading. Just a quick summary of the issues I'm percieving... Client side filtering allows for more individual customization. Individualized Corpuses are needed, but this then puts the bandwidth burdon on the indivisual. They STILL have to pull down the spam from the POP server. But this POP server can also have filtering taking place as well. Is it my understanding that there is to be a more generalized type of filtering, applicable to the general group of Email users sharing the same POP server? Then having a more customizable filtering take place (either through processing scripts on the POP serer, or on Client level code, running as a POP Proxy, or as some other localized stript needed for filtering. Most of all this theorical statistical discussions are far above my head, and the lack of my old College Staatistical textbooks where are probably outdated anyway, I decided to take a less focus on these discussions, and want to contribute by helping make our efforts more usable to the spam fighting community. My experience in programming, is more to throw something together to get a "proof of permance" system up and running. My best talants are to "organize" different "Object" modules so they can be used for the largest range of applications as possible. Let me throw in some really cool concepts I learned, which i feel sort of "attached" to. I had an opportunity to use Symantic's Think Class Liberty, a collection or wuite of powerful C++ object that define computer functionality analygous to the way "People" work. Let me give you an example... Lets say we have a dialog box, with numerous emdbedded and hierarchical "views". One can think of the overall dialog as an "organization", not onlike an "Accounting" department, or "Development" department. So, in effect, the Dialog would be a "supervisor" to raign over each of the elements inside. But each of these elements can also be "supervisors" over things inside them. NOT to be confused with "Inheritance" hierarchy, there is a "Command" hierarchy. The rule is simple... each of the "Subordinate Bureaucrats" as I like to call them (name comes from Think Class Library), never has to know anything about it's supervisor. It is the supervisor's responsibility to manage and "inform" the collection of objects under it's "rule" or "control". A "bureaucrat", which is derived from "Object" has just two instance variables. One is a pointer to it's supervisor, and the other is a pointer to a List or Collection object, also pointing to other bureaucrats in the organization. If I subordinate does something, that other subordinate objects have to "know about", it sends a "DoCommand" message with an enumerated command number (Known ONLY to the supervisor, would then delegate control up to the Supervisor, which would then know what to do, like notify all the other subordinate objects if appropriate. I propose that I come up with a list of "Bureaucrats" as I like to call them, not unlike what might be desired in a "mailroom". Then define Spam fighting Bureaucrats that can work together on ANY machine, and allow people to easily put together a host or Object Class Library of not just spam fighting "agents", but also "mail handling" functions. I'm making up a list now, of bureaucrats I would want to see this have. When I'm done with this list, I would like to publish it to the group for discussion. In meantime, start sending me your nominations for "bureaucrats", and I can start a list of them. Start with what you've done alreay (in the way of Object classes that can be turned into bureacrats), define their relationships, then group them into "agencies" or collections of bureaucrats needed for whatever function appropriate. I'll throw out some "teasers". We need a "Configurator" bureaucrat, that all it does is manage the server or client's "comfiguration". Some people find configuration files convenient to manage their configurations, others profer GUI's of local or web based types. The Configurator would store it's objects in some persistant manner, plus be able to read in a "configuration" file and setup that way. I'm betting this is already being done, but with different names. So I'm proposing we collect these objects not unlike what a "human" organization would have to be, to manage someone's mail. Python already has these... "Message", and a few others. We just make "Message" have an additional parent of a Bureaucrat. A "Corpus manager" would manage a "corpus" and have all the smarts necessary to do that. A "Tokenizer" would manage and control all the "Token" objects. Another one would be Josiah Carlson's "PASP" which would derive from a "POP3" object (of course also deriving from Bureaucrat). Anyway, I ask for your comments on this approach to defining the "objects" needed to make all this useful. John

3 3

Patch and info on how to run a test
by papaDoc Oct. 1, 2002

Oct. 1, 2002

Hi, This is a small patch to help people which don't have python in their path. They will need to change runtest.sh at only one place. runtest.sh is talking about # This test requires you have an appropriately-modified # Tester.py.new and classifier.py.new as detailed in # <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one(a)comcast.net> Where can I find those two files ? There is no search on the mailing list archive and I subscribed to the list after this message. Also I can run some test but I need to a good .ini file can someone provide me with one. I don't have 2000 spams yet only 1546 now but going up every days. papaDoc Index: runtest.sh =================================================================== RCS file: /cvsroot/spambayes/spambayes/runtest.sh,v retrieving revision 1.4 diff -r1.4 runtest.sh 31a32,33 > PYTHON=python > 34,35c36,37 < python rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n 0 -Q < python rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n 0 -Q --- > $PYTHON rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n 0 -Q > $PYTHON rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n 0 -Q 37,38c39,40 < python rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n $RNUM -Q < python rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n $RNUM -Q --- > $PYTHON rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n $RNUM -Q > $PYTHON rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n $RNUM -Q 43c45 < python timcv.py -n $SETS > run1.txt --- > $PYTHON timcv.py -n $SETS > run1.txt 46c48 < python timcv.py -n $SETS > run2.txt --- > $PYTHON timcv.py -n $SETS > run2.txt 48c50 < python rates.py run1 run2 > runrates.txt --- > $PYTHON rates.py run1 run2 > runrates.txt 50c52 < python cmp.py run1s run2s | tee results.txt --- > $PYTHON cmp.py run1s run2s | tee results.txt 57c59 < python timcv.py -n $SETS > run1.txt --- > $PYTHON timcv.py -n $SETS > run1.txt 63c65 < python timcv.py -n $SETS > run2.txt --- > $PYTHON timcv.py -n $SETS > run2.txt 65c67 < python rates.py run1 run2 > runrates.txt --- > $PYTHON rates.py run1 run2 > runrates.txt 67c69 < python cmp.py run1s run2s | tee results.txt --- > $PYTHON cmp.py run1s run2s | tee results.txt 80c82 < python timcv.py -n $SETS > run1.txt --- > $PYTHON timcv.py -n $SETS > run1.txt 90c92 < python timcv.py -n $SETS > run2.txt --- > $PYTHON timcv.py -n $SETS > run2.txt 92c94 < python rates.py run1 run2 > runrates.txt --- > $PYTHON rates.py run1 run2 > runrates.txt 94c96 < python cmp.py run1s run2s | tee results.txt --- > $PYTHON cmp.py run1s run2s | tee results.txt

2 1

memory consumption...
by Anthony Baxter Oct. 1, 2002

Oct. 1, 2002

For my standard test, using the graham scheme, memory consumption's blown out considerably in the last couple of days. timcv -n 10 used to consume about 50M - it's now about 75M. This unfortunately pushes my system into swap :-( is there anything obvious to change to make it happy again?

7 9