Mailman 3 February 2004 - spambayes-dev

A Question about SpamBayes
by Joachim Engelhardt Feb. 19, 2004

Feb. 19, 2004

Hi there, first of all, SpamBayes is GREAT!!! I have it running now since two or three weeks and I love it. Fantastic work. However, I am missing one tiny little feature, which you are addressing under 4.3 and 6.5 in your FAQ section: Return/bounce/forward spam back to the sender. In FAQ 4.3 it sounds like that this is somehow possible. But, I have no clue what are you talking about in there... By the way, I am running the Outlook (2000) plugin under Windows 200 Pro - no Exchange server. So is it possible now or not? In FAQ 6.5 it is stated that I can't bounce spam back to the sender since most sender addresses are fake anyway. I am in full agreement with you on that and go along with this statement totally. However, there are always messages that get filtered out by SpamBayes that are not spam and are legitimate. Therefore, I am always browsing over the Junk mail folder before deleting all the spam - making sure I am not deleting an important message. Now, if I could autoreply to all messages in the Junk E-Mail folder and attach a short message to it, at least the non-spam senders would be notified automatically that their email was considered spam and has not been read but deleted. This way they could try to resend or rephrase. The return address from non-spammers should be a good one - and the autoreplies to spammers with fake return addresses end up in limbo. Do you think this makes any sense and are you considering implementing something like that into SpamBayes? Thanks, Joe

2 2

RE: [spambayes-dev] getting rid of the "new mail" icon
by antoine.trux＠nokia.com Feb. 19, 2004

Feb. 19, 2004

> - Ignore the new-mail icon. That's what I do. In fact, I started > ignoring it long before SpamBayes existed -- its behavior never > made useful sense to me. "Tell us what you need, we will explain why you don't need it."

2 1

RE: [spambayes-dev] getting rid of the "new mail" icon
by antoine.trux＠nokia.com Feb. 19, 2004

Feb. 19, 2004

> > Can you confirm that it is not currently possible to configure > > SpamBayes for usage scenario 1? > > I can confirm that this not currently possible. Well, then I am left with the following options: - Switch back to Spammunition (which does switch off the new mail icon). - Use a commercial anti-spam filter (if I can find one that switches off the new mail icon; suggestions welcome). - Change my email address. I am thrilled by none of these possibilities. I would have liked to use SpamBayes, but given the amount of spam I get, this is just not possible. Antoine

4 4

RE: [spambayes-dev] Mozilla SpamBayes "porting"
by Tony Meyer Feb. 19, 2004

Feb. 19, 2004

> The corpus I used has 2,793 hams and 948 hams. I presume one of those is spam, or you've got amazing results <wink>. If your testing setup allows it, have a go with 948 of each, and see what that does. > We're using SpamAssasin's public corpus for testing. In case it helps, last time I ran timcv with my copy of the SA corpus, split into 5 groups, I got: -> <stat> tested 655 hams & 2389 spams against 2624 hams & 9557 spams filename: sa_base ham:spam: 3279:11946 fp total: 7 fp %: 0.21 fn total: 85 fn %: 0.71 unsure t: 359 unsure %: 2.36 real cost: $226.80 best cost: $187.00 h mean: 0.81 h sdev: 6.74 s mean: 98.20 s sdev: 10.32 mean diff: 97.39 k: 5.71 (This is with all defaults). If you've got the corpus lying around in one-text-file-per-email format, then the easiest way to test would be to install Python and SpamBayes, and run timcv.py over the corpus and see if the results you get look something like the Mozilla ones (I suppose you could do -n2 to simulate splitting the corpus in half, rather than more sets as is common here). > It's kind of tough to test since we don't have the nice > cross-validation tools that you have You could write some, of course <wink>. =Tony Meyer

2 1

RE: [spambayes-dev] Mozilla SpamBayes "porting"
by Tony Meyer Feb. 19, 2004

Feb. 19, 2004

> I'd really love it if Moz/Thunderbird's spam filtering were > pluggable. +1 (ok, so this is the wrong list for this...) =Tony Meyer

1 0

RE: [spambayes-dev] Mozilla SpamBayes "porting"
by Tony Meyer Feb. 19, 2004

Feb. 19, 2004

[Miguel] > Here is the core of my C++ port if anyone wants to take a look. > You'll notice that I included the > "experimental_ham_spam_imbalance_adjustment", could this be > my problem? [Adam] > Usage suggests that "experimental_ham_spam_imbalance_adjustment" > produced bad results for most people. It is no longer used by > most people. And in fact the code doesn't even exist in current CVS or in 1.0a9 (0.9), even though the option is there (but deprecated). So only people using out-of-date spambayes might be using it. Note that this could only effect your results if you did in fact have an imbalance - you didn't say how the corpus was split. =Tony Meyer

2 1

RE: [spambayes-dev] Problem with 1.0a9 Windows installer?
by Tony Meyer Feb. 19, 2004

Feb. 19, 2004

> Yep, this was the fix that I needed to make it work for me. > I also solved the mystery of why pythoncom.frozen isn't set, > although I'm still clueless why it worked in some cases and > not in others. Yeah, I wish I knew that, too >:( > Here's the reason why pythoncom.frozen isn't set: [...] I think this is correct, though. Since we're the one pretending to be a dll, we should be the one to set pythoncom.frozen, too. No doubt Mark will soon weigh in with his wisdom, and then we'll all understand <wink>. =Tony Meyer

1 0

getting rid of the "new mail" icon
by antoine.trux＠nokia.com Feb. 18, 2004

Feb. 18, 2004

Hi, I just installed the SpamBayes plug-in for Outlook 2000. SpamBayes seems superior in all respects to the previous spam filter I was using (Spammunition, see www.upserve.com ), except for one important detail: I was able to configure Spammunition so that it would switch off the "new mail" (envelope shaped) icon after detecting a spam, but I could not manage doing it with SpamBayes. According to http://spambayes.sourceforge.net/faq.html#how-can-i-get-rid-of-the-envelope… , this functionality would be very hard to implement. This FAQ item says: "This means that even if you have set SpamBayes to mark spam as read, the envelope tray icon will not vanish." However, the manual describes option spam_mark_as_read as follows (C:\Program Files\SpamBayes\docs\outlook\docs\configuration.html): "Determines if spam messages are marked as 'Read' as they are filtered. This can be set to 'True' if the new-mail icon bothers you when the only new items are spam." This does not seem to work for me. Consider this scenario: 1) My "Inbox" folder is empty (yes, absolutely empty). No "new mail" icon. 2) A spam arrives and is detected by SpamBayes. The "new mail" icon is on. As I wrote above, it is possible to configure Spammunition so that the "new mail" icon be switched off in this same scenario. Now, this same FAQ item (http://spambayes.sourceforge.net/faq.html#how-can-i-get-rid-of-the-envelope…) says: "Although there is code available that provides a method to delete this icon, it doesn't let us determine whether there is other unread mail as well, which means that we do not know whether we should delete the icon or not." So you prefer not to switch off the icon if you are not sure. It would be most valuable, however, if SpamBayes could switch off the "new mail" icon each and every time a spam message is detected, or at least provide an option to work that way. This is because I much prefer: 1) not to be constantly distracted by the "new mail" icon than: 2) having to check my Inbox every time the "new mail" icon is switched on (which happens every 5th minute or so for me, because I get about 250 spams a day). With usage scenario 1, I can work this way: - When I hear the new message's sound, I have a look at the "new mail" icon: - Most of the time, the icon is off (because the vast majority of the messages I get are spams and are detected by SpamBayes). - Sometimes, the icon is on. I then know that a ham has arrived (or a spam that SpamBayes does not detect), so I can interrupt my work. - After being absent and returning to my desk, I immediately check whether I have new hams even if the "new mail" icon is off, because I know it could have been switched off by SpamBayes after detecting a spam. Can you confirm that it is not currently possible to configure SpamBayes for usage scenario 1? Antoine

2 1

Understanding Classifier Code
by Daniel Eloff Feb. 18, 2004

Feb. 18, 2004

I've been looking at ways of increasing the speed at which the classifier runs. I think that increasing the speed of the classifier would be one of the things required to make spam-bayes computationally worthwhile to run on a gateway for several thousand users. (others include keeping all word records in memory, and writing the really tight parts into C or assembler) No arguments yet please, I just need some help to understand what the classifier is doing and why. I'll paste the code and interject when i don't understand why something is happening that way. (I don't like to modify code without understanding it fully, you're asking for trouble if you do that.) I really appreciate any help you can give me on this, the more the merrier. And if my team manages to get spambayes working well on our server we'll be sure to share the modifications with you. H = S = 1.0 Hexp = Sexp = 0 clues = self._getclues(wordstream) for prob, word, record in clues: S *= 1.0 - prob H *= prob if S < 1e-200: # prevent underflow S, e = frexp(S) Sexp += e if H < 1e-200: # prevent underflow H, e = frexp(H) Hexp += e Tell me, why a seperate spam/ham score at this point? # Compute the natural log of the product = sum of the logs: # ln(x * 2**i) = ln(x) + i * ln(2). S = ln(S) + Sexp * LN2 H = ln(H) + Hexp * LN2 Okay i can see from the comment that this is equiv to ln(x * 2**i) But why take the logarithim of the final spam/ham prob? n = len(clues) if n: S = 1.0 - chi2Q(-2.0 * S, 2*n) H = 1.0 - chi2Q(-2.0 * H, 2*n) Why multiply the score by -2? why double n? def chi2Q(x2, v, exp=_math.exp, min=min): """Return prob(chisq >= x2, with v degrees of freedom). v must be even. """ assert v & 1 == 0 # XXX If x2 is very large, exp(-m) will underflow to 0. m = x2 / 2.0 sum = term = exp(-m) for i in range(1, v//2): term *= m / i sum += term What's going on here? Why do we take the exp() of -x2/2? Why did we multiply x2 by 2 if we divide it by 2 again anyway? What is the loop doing and why? # With small x2 and large v, accumulated roundoff error, plus error in # the platform exp(), can cause this to spill a few ULP above 1.0. For # example, chi2Q(100, 300) on my box has sum == 1.0 + 2.0**-52 at this # point. Returning a value even a teensy bit over 1.0 is no good. return min(sum, 1.0) Thanks again! _________________________________________________________________ Help STOP SPAM with the new MSN 8 and get 2 months FREE* http://join.msn.com/?page=dept/bcomm&pgmarket=en-ca&RU=http%3a%2f%2fjoin.ms…

3 2

Understanding Classifier Code
by Daniel Eloff Feb. 18, 2004

Feb. 18, 2004

I've been looking at ways of increasing the speed at which the classifier runs. I think that increasing the speed of the classifier would be one of the things required to make spam-bayes computationally worthwhile to run on a gateway for several thousand users. (others include keeping all word records in memory, and writing the really tight parts into C or assembler) No arguments yet please, I just need some help to understand what the classifier is doing and why. I'll paste the code and interject when i don't understand why something is happening that way. (I don't like to modify code without understanding it fully, you're asking for trouble if you do that.) I really appreciate any help you can give me on this, the more the merrier. And if my team manages to get spambayes working well on our server we'll be sure to share the modifications with you. H = S = 1.0 Hexp = Sexp = 0 clues = self._getclues(wordstream) for prob, word, record in clues: S *= 1.0 - prob H *= prob if S < 1e-200: # prevent underflow S, e = frexp(S) Sexp += e if H < 1e-200: # prevent underflow H, e = frexp(H) Hexp += e Tell me, why a seperate spam/ham score at this point? # Compute the natural log of the product = sum of the logs: # ln(x * 2**i) = ln(x) + i * ln(2). S = ln(S) + Sexp * LN2 H = ln(H) + Hexp * LN2 Okay i can see from the comment that this is equiv to ln(x * 2**i) But why take the logarithim of the final spam/ham prob? n = len(clues) if n: S = 1.0 - chi2Q(-2.0 * S, 2*n) H = 1.0 - chi2Q(-2.0 * H, 2*n) Why multiply the score by -2? why double n? def chi2Q(x2, v, exp=_math.exp, min=min): """Return prob(chisq >= x2, with v degrees of freedom). v must be even. """ assert v & 1 == 0 # XXX If x2 is very large, exp(-m) will underflow to 0. m = x2 / 2.0 sum = term = exp(-m) for i in range(1, v//2): term *= m / i sum += term What's going on here? Why do we take the exp() of -x2/2? Why did we multiply x2 by 2 if we divide it by 2 again anyway? What is the loop doing and why? # With small x2 and large v, accumulated roundoff error, plus error in # the platform exp(), can cause this to spill a few ULP above 1.0. For # example, chi2Q(100, 300) on my box has sum == 1.0 + 2.0**-52 at this # point. Returning a value even a teensy bit over 1.0 is no good. return min(sum, 1.0) Thanks again! _________________________________________________________________ Dream of owning a home? Find out how in the First-time Home Buying Guide. http://special.msn.com/home/firsthome.armx

1 0