Mailman 3 June 2004 - spambayes-dev

Mental Musings on Spam Catching
by Stephen Anderson July 3, 2004

July 3, 2004

[Right off the bat, my email is long-winded and I apologize. I think Ben Franklin once said something like, "I apologize but I don't have time to be brief."] I've been slowly trying to design a spam filter program using many of the lessons learned tuning spambayes as a starting place. I'm not saying I tuned spambayes but I've followed most of the discourses on the list and read through the source documentation for several ideas that predated my joining. I'm going to give you a little background (cause somebody always asks) but I was hoping maybe some of you with more experience could give me your opinion on a filtering idea that I had. (I know, theorizing is nothing, testing is everything) There are really four reasons I've been trying to do this. 1) It's fun and I like it; 2) My gut tells me SB is missing a lot of potentially useful information; 3) SB algorithm development has really slowed down from its heyday (which I can understand since it seems to work for most of the developers) and; 4) SB really underperforms on my father's email in comparison to most other peoples, my own included. That last point isn't so much why I started trying to build my own but has been leading me to try to figure out what exactly is SB's weakness in that it is so unsure (and so often wrong) about his email. The general direction I had been working towards is something very SB'ish with three main exceptions: 1) much more reliant on meta-tokens for email characteristics (ala SA) than SB, 2) Very different token scoring using mean and std-dev of occurrence / # of total body tokens; and 3) automatic expiration and removal of statistics after a certain time-period. Now a facet that I had been thinking about was elimination of duplicate messages. In the system I'm building, I don't want the system to ever be trained on the same message more than once. But obviously I don't want to store every message it has seen up to the expiration date (which I'm thinking would be 9-16 months). I also wanted to eliminate substantially duplicate emails. My googling brought me to the Distributed Checksum Clearinghouse and DCC's use of a "fuzzy checksum" algorithm to do almost exactly what I was looking for. After discovering that, I started contemplating how else I might incorporate a fuzzy checksum. I hadn't really thought of anything, but it put me in the right frame of mind for my next thought. I was reading through Tim Peters' very recent list response about "Correlated Values." I have reflected a lot on that very subject. How in the world can the system know that some highly correlated values should have more weight than other highly correlated values? Honestly, I've never been able to come up with any reasonable scheme that might work. I was sitting on my think chair (flush..) contemplating all that when I asked myself how some small emails could be so obviously spam to me and yet be so hard for SB because they have a few highly correlated ham tokens and fewer highly correlated spam tokens. I realized that its much easier for me because I'm looking at the whole picture, not on a token by token basis. So this brought me round to the bi-grams and n-grams thinking. I'm no real fan of them because I don't like the database growth and past poor tested performance. I was also mulling over DSPAM's n-gram offshoot scheme which I can't remember the name of. Finally it all kind of coalesced in my mind to the idea I wanted to run past everybody. Flavors. Emails have flavors and you can mix and match flavors. Some mixes lead to good tasting emails and some lead to bad tasting emails. Just like desserts. I can have ice cream, nuts, fruit, chocolate syrup, sugar sprinkles, and other stuff. Ice cream, fruit, and chocolate syrup is usually a winner. Nuts, sugar sprinkles, and chocolate syrup is usually not a good desert. What if we could use n-gram statistics to process the email as a more gestalt entity. But, instead of using n-grams on the words, what if we used it on a fixed set of potential tokens. I was thinking how one might take the token stream from the body and come up with a fuzzy checksum if you will. But instead of going the fuzzy checksum route, could we analyse the token stream to come up with meta-tokens that would indicate a more complex characteristic of the email than just a single word or token could provide? For instance, what if after all the tokens are generated, we can look at the token stream and perform some defined analysis to make decisions about certain specific characterstics of email messages. I know the devil is in the details and I haven't thought that far ahead. Right now I'm thinking things like : HIGH-PRCNT-HTML, or TYPICAL-MAILLIST, or HIGH-PIC- CONTENT, or NESTED-ENCODING-SCHEMES, or HIGH-HAPAXE-CONTENT, or HIGH- CODE-CONTENT or HAM-BLOCKS-AND-SPAM-BLOCKS. There should be characteristics we could objectively define about email messages where the characterstics are orthogonal and each one conveys information aggregated from the token stream but more than any single regular token could convey. Most of my examples are crap, but the high mailing list probability is a good one. There should be several email traits that we could see in the token stream that when taken together are characteristic of a mailing list message. This could then be one flavor token. After this, we can use an n-gram scheme to give every email message a flavor list. Like this message I'm writing could be FLAVS-A,D,E where A is mailing list content, D is long-winded, and E is high sentence length. With 10 email characteristics there would be 1023 possible combinations. That's not a lot to track in the database but I think it would be very valuable because certain flavor combinations will be very spammy and certain ones will be very hammy. So many of my father's emails that SB stumbles on are tripped up because spammers (or dumb luck) have been able to put highly-correlated ham words inside with their short spam message. But these emails don't have the same flavor of a real ham message. I guess what I'm trying to say is that it's relatively easy to bury spam in fake ham "content" because we look at the words in isolation. It would be considerably harder to fake multiple characteristics that consider the email in its entirety. Okay, I should sleep more and type less. Thanks for your time. Cheers, Steve

2 1

Re: [spambayes-dev] correlated clues
by Toby Dickenson July 2, 2004

July 2, 2004

On Monday 28 June 2004 04:24, Tim Peters wrote: > [Toby Dickenson] > > > Im seeing a significant number of misclassified spams that come through > > mailing lists. > > Toby, which training strategy do you use? As you guess, I train on everything. Its very low effort to maintain, but it has left me with several training imbalances like this that have adversely affected classification accuracy. > A general problem with "doing something about this" is that correlation > *usually* helps in determining a correct classification, so I don't think > correlation is evil per se. > I don't have a good idea for identifying "bad correlation" > automatically and efficiently. My standards are lower... For now I would happy with a manual and inefficent process that picked out just this one special type of correlated clue. A couple of prototypes later, and even that isnt as easy as I hoped :-( Ill let you know if anything good come this. -- Toby Dickenson

5 6

Automatische Antwort - Newsletter Lifestyle
by Lyris ListManager June 30, 2004

June 30, 2004

Liebe User, dies ist eine automatisch erzeugte Mail. Wenn Sie Ihren Newsletter abbestellen wollen, müssen Sie auf den »Abbestellen«-Button am Ende des Newsletters klicken. Achtung: Für diesen Prozess müssen Sie online sein. Sollte Ihr Mail-Client den Button deaktiviert haben, können Sie folgende Seite aufrufen: http://www.stern.de/mein-stern-de/newsletter/ Im Kasten der rechte Spalte markieren Sie den Newsletter, den Sie nicht mehr erhalten möchten, und geben danach Ihre E-Mail-Adresse ein. Sie werden sofort vom Verteiler entfernt. Ihr stern.de-Team.

1 0

Saturday 3rd July
by Tina Jones June 29, 2004

June 29, 2004

1 0

RE: [Spambayes] Execute test suite
by Kenny Pitt June 29, 2004

June 29, 2004

Tony Meyer wrote: >> This screenshot is taken from a development version running from >> source code. SpamBayes detects if it is running from the released >> binaries or from source code, and it does not show the "Execute test >> suite" option if running from the released binaries. > > We really ought to use a shot from the binary and use that. Any > chance you want to do that? <wink> OK, I grabbed the attached screenshot. I'm running Outlook 2003 so the toolbar style is a little different. If that's OK with everyone then I'll go ahead and check it in on the trunk and Tony can migrate it back to the 1.0 branch if desired. -- Kenny Pitt

2 1

RE: [spambayes-dev] RE: Difference between "show clues" scoring andfilter scoring
by Tony Meyer June 29, 2004

June 29, 2004

> I'd be curious to know even why it scored 15 for Tony! > There's nothing clearly spammy about it, apart from a > Yahoo blurb at the bottom, and yahoo sender address. I do get a bit of spam from Yahoo (lists), so that's part of it. The other spammy clues are all from 6 or fewer messages, so maybe 'small database' accounts for some of it. The imbalance probably doesn't help (though doesn't the imbalance in this direction more account for false negatives?). Combined Score: 15% (0.150615) Internal ham score (*H*): 0.802823 Internal spam score (*S*): 0.104054 # ham trained on: 55 # spam trained on: 322 50 Significant Tokens token spamprob #ham #spam 'bi:2000 and' 0.0918367 2 0 'bi:outlook 2003' 0.0918367 2 0 'outlook' 0.108078 4 2 'it.' 0.116936 10 7 'filter' 0.135074 3 2 'bi:they both' 0.155172 1 0 'skip:_ 30' 0.155172 1 0 'built' 0.164747 4 4 'spam' 0.164747 4 4 'anything' 0.173132 7 8 'there' 0.186543 22 29 'settings' 0.210929 1 1 'upgraded' 0.210929 1 1 "i've" 0.218646 7 11 'check' 0.257231 14 28 'skip:_ 40' 0.260614 6 12 'use' 0.261442 19 39 'url:listinfo' 0.270438 8 17 'url:mailman' 0.270438 8 17 'before' 0.271457 13 28 'on.' 0.271748 2 4 'loved' 0.286634 1 2 'url:org' 0.288353 17 40 'need' 0.289067 14 33 'order' 0.289706 6 14 'yahoo!' 0.294234 3 7 'bi:url:mail url:python' 0.329461 6 17 'skip:a 10' 0.336552 25 74 'subject:] ' 0.339934 16 48 'url:sf' 0.344635 3 9 'header:Errors-To:1' 0.360176 21 69 'subject:Spambayes' 0.371204 7 24 'sender:no real name:2**0' 0.371437 20 69 'url:yahoo' 0.377219 4 14 'more' 0.38184 23 83 'subject:[' 0.390077 15 56 'bi:header:Received:6 header:From:1' 0.393314 9 34 'sender:addr:spambayes-bounces' 0.397277 6 23 'url:html' 0.603261 12 107 'bi:header:MIME-Version:1 proto:http' 0.62941 4 40 'bi:been using' 0.844828 0 1 'bi:than other' 0.844828 0 1 'bi:thank you,' 0.844828 0 1 'bi:subject:? subject: ' 0.934783 0 3 'subject:2003' 0.934783 0 3 'understand' 0.934783 0 3 '2003,' 0.949438 0 4 'subject:with' 0.958716 0 5 'subject:\n\t' 0.965116 0 6 'from:addr:yahoo.com' 0.987106 0 17 =Tony Meyer

1 0

Difference between "show clues" scoring and filter scoring
by Tony Meyer June 29, 2004

June 29, 2004

I just had a very odd experience - a message (the one that Tim just replied to on spambayes(a)python.org) arrived and ended up in my spam folder. The spam field has a value of 1.00 in it - I was quite surprised at the false positive, so I did a "Show clues", and it scores 15% (high, but ok). I can't figure out why there is a discrepancy. There was definitely no training (I even checked the log, which confirms the move to the spam folder) between the two scorings. Does "show clues" get the message differently somehow? Is there some other explanation for this? FWIW, there aren't any errors in the log, either. =Tony Meyer

3 2

RE: [spambayes-dev] RE: Difference between "show clues" scoring andfilter scoring
by Tony Meyer June 29, 2004

June 29, 2004

> If the message you are refering to is a mime message, > it may be the same thing. I believe the issue will be > the butchery we do with the content-type and mime-armour. The message has a "MIME-Version" header, but no other mime headers. I've attached the message to the tracker. > Unless I am mistaken, everything after the blank line > is part of the body - but as I mentioned, Outlook is > returning it in the headers. I assume, but have not > verified, that us fetching the headers via the MAPI > property will give us the same string. I also suspect, but > haven't confirmed, that this will screw up what we do > with the headers, especially if we append headers > *after* the blank line. Does that still fit with my message that didn't have the multipart stuff? =Tony Meyer

1 0

FAQ quest? Using spam bayes with outlook 2003 spam filter
by Kim Stockdale June 29, 2004

June 29, 2004

I've been using spam bayes with Outlook 2000 and loved it. I upgraded to Outlook 2003, and I understand Outlook's built in spam filter is automatically on. Is there anything I need to do to the Outlook 2003 spam settings in order to use Spam Bayes? Can/should they both be used? Thank you, Kim Stockdale __________________________________ Do you Yahoo!? Yahoo! Mail - 50x more storage than other providers! http://promotions.yahoo.com/new_mail

1 0

RE: [spambayes-dev] RE: [Spambayes] Execute test suite
by Tony Meyer June 28, 2004

June 28, 2004

> -----Original Message----- > From: spambayes-dev-bounces(a)python.org > [mailto:spambayes-dev-bounces@python.org] On Behalf Of IIMS > Postgraduate Representative [...] Opps. Don't clean your keyboard while in the middle of typing message, or look what might happen <wink>. =Tony Meyer

1 0