Mailman 3 February 2004 - spambayes-dev

inboxer ad
by Skip Montanaro Feb. 23, 2004

Feb. 23, 2004

Hey folks, I just noticed an Inboxer ad on SF. Maybe this is old hat (I was on vacation last week). Thought I'd pass it along though. Skip

1 0

Mozilla SpamBayes "porting"
by Miguel Feb. 23, 2004

Feb. 23, 2004

Sorry for the semi-offtopic post, but any help will be very apreciated. Mozilla's mail apps currently use Paul Graham's original algorithm with some basic tokenizing techniques. This situation could use some improvement, so now there is an effort to copy some ideas from Spambayes into Mozilla. I wrote a Mozilla patch that tries to port the chi2-combining techniques from classifier.py into Mozilla's C++. My testing is showing huge improvements in the fn rates, but a big deterioration in the fp rates. For example, in a test with a 3,741 email corpus we got: original - fn:206 fp:0 chi2 patch - fn:63 fp:11 My question is, did you guys notice a similar increase in fp rates when you originally switched from Graham to chi2? If not, then I'll assume that I made a mistake in porting classifier.py. Many thanks, Miguel PS. If anyone is interested in what Mozilla is doing, you can look here: http://bugzilla.mozilla.org/show_bug.cgi?id=181534 http://bugzilla.mozilla.org/show_bug.cgi?id=230093 http://bugzilla.mozilla.org/show_bug.cgi?id=231873 Here is the core of my C++ port if anyone wants to take a look. You'll notice that I included the "experimental_ham_spam_imbalance_adjustment", could this be my problem? double spam2ham = dmin(nbad/ngood, 1.0); double ham2spam = dmin(ngood/nbad, 1.0); /** This section comes from probability(self, record) and _getclues(self, wordstream)**/ for (i = 0; i < count; ++i) { Token& token = tokens[i]; // tokens is an array of Token, elements of a Token // include both token.mProbability and token.mDistance const char* word = token.mWord; Token* t = mGoodTokens.get(word); double hamcount = ((t != NULL) ? t->mCount : 0); t = mBadTokens.get(word); double spamcount = ((t != NULL) ? t->mCount : 0); prob = (spamcount / nbad) / ( hamcount / ngood + spamcount / nbad); double n = hamcount * spam2ham + spamcount * ham2spam; prob = (0.225 + n * prob) / (.45 + n); double distance = abs(prob - 0.5); if (distance >= .1) { goodclues++; token.mDistance = distance; token.mProbability = prob; } else { token.mDistance = -1; //ignore clue } } // sort the array by the token distances PRUint32 first, last = count; if (count > 150) { first = count - 150; // This function sorts the array by token.mDistance NS_QuickSort(tokens, count, sizeof(Token), compareTokens, NULL); } else { first = 0; } /** This section comes from chi2_spamprob(self, wordstream, evidence=False) **/ double H = 1.0, S = 1.0, Hexp = 0, Sexp = 0; goodclues=0; int e; for (i = first; i < last; ++i) { if (tokens[i].mDistance != -1) { goodclues++; double value = tokens[i].mProbability; S *= (1.0 - value); H *= value; if ( S < 1e-200 ) { S = frexp(S, &e); Sexp += M_E; } if ( H < 1e-200 ) { H = frexp(H, &e); Hexp +=M_E; } } } S = log(S) + Sexp * M_LN2; H = log(H) + Hexp * M_LN2; if (goodclues>0) { S = 1.0 - chi2Q(-2.0 * S, 2 * goodclues); H = 1.0 - chi2Q(-2.0 * H, 2 * goodclues); prob = (S-H +1.0) / 2.0; } else { prob = 0.5; } PRBool isJunk = (prob >= 0.90); //hardcoded at .9 ------------------------------------ Here's the chi2Q funcition: static double chi2Q (double x2, double v) { PRUint32 i; double m = x2 / 2.0; double sum = exp(-m); double term = exp(-m); for (i=1;i<=floor(v/2);i++) { term *= m / i; sum += term; } return dmin(sum,1.0); }

7 16

Mac OS X package
by Sam Thorne Feb. 22, 2004

Feb. 22, 2004

Hello all, New to the list, and pretty new to spambayes. I've put together a package for easy installation of spambayes on Mac OS X. Posted it on the wiki; http://www.entrian.com/sbwiki/MacOSXPackage Generally wondering a few things though; in the install I've created, I put everything into a spambayes folder in the Library; this is the 'nice' place to install things on Mac OS X so they are self-contained and easy to find/uninstall. The pop proxy works fine here, with the folder structure being: /Library/SpamBayes/(contents of Scripts folder from source + data files) /Library/SpamBayes/spambayes/ (contents of the spambayes folder from source) However, the utilities (such as which_db etc) still look for the libs in /System/Library/Python.framework/... and don't find them (obviously). In the end I left them out, as the point of the package really was just to create a very easy point 'n' click install, so I doubt many will use the utilities, but I'm still wondering why everything else checks the current directory first and then elsewhere for it's modules etc. when the testtools and utilities don't...? And also, am I breaching the license by leaving them out? Anyway, hopefully that's all ok but I'd appreciate any feedback on the pkg. -- Sam So long, and thanks for all the fish.

1 0

Some samples that fool spambayes
by Jean-Marc Valin Feb. 22, 2004

Feb. 22, 2004

Hi, In the last few days, I've been hit with spam that got past through spambayes. This spam, which seems to come from the same spammer, always gets through because the spammer includes lots of "hammie" words at the end. I thought it would be interesting to add these messages to your test setup because they're good examples of spam actively trying to defeat a filter. I've posted the mbox file here: http://www.speex.org/~jm/mbox Jean-Marc P.S. I'm not on the list, so please CC to me on your reply. -- Jean-Marc Valin, M.Sc.A., ing. jr. LABORIUS (http://www.gel.usherb.ca/laborius) Université de Sherbrooke, Québec, Canada

1 0

RE: [PSF-Board] Spambayes
by Randy Kondor Feb. 22, 2004

Feb. 22, 2004

Folks, can you please contact me about making some enhancements to Spambayes? I can provide funding for this work. Your product is awesome!!! Regards, Randy Matrikon - We make connections! Watch the multimedia OPC Tutorial at http://www.matrikon.com/tutorial O-----------------------------------------------------------------O | Randy Kondor Phone: 780-945-4035 | | OPC Product Manager Fax: 780-448-9191 | | Matrikon Email: randy.kondor(a)matrikon.com | | 10405 Jasper Ave, Suite 1800 WEB: http://www.matrikon.com | | Edmonton, Alberta, Canada, T5J 3N4 | O-----------------------------------------------------------------O Attend Matrikon's Annual User Group May 2-5, 2004 Matrikon Valued Partners Event - MVP 2004 Driving Performance, Partners for Results. -----Original Message----- From: Tim Peters [mailto:tim.one@comcast.net] Sent: Friday, February 20, 2004 11:56 AM To: Randy Kondor Cc: psf(a)python.org Subject: RE: [PSF-Board] Spambayes [Randy Kondor] > I am interested in speaking with you about funds for Spambayes. > Please call me at 780-945-4035. Randy, the SpamBayes developers chose to give copyright to the PSF, and to release the code under the PSF license, but the PSF has no other connection with that project. For example, the PSF doesn't fund it, or manage it. You can reach the SpamBayes developers directly via: mailto:spambayes-dev@python.org All SpamBayes developers are currently unpaid volunteers, but there's no prohibition against them doingwork for pay. There's also no prohibition against using the SpamBayes code in a commercial project (and, for example, at least http://www.inboxer.com/ has done so). **************************************************************************** * READER BEWARE: Unencrypted, unsigned Internet e-mail is inherently insecure. Internet messages may be corrupted, incomplete, misdirected or may incorrectly identify the sender. Therefore, nothing in this message or attachments may be considered legally binding. THIS MESSAGE IS ONLY INTENDED FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY BE PRIVILEGED. If you are not the intended recipient or their authorized agent, you may not forward or copy this information and must delete or destroy all copies of this message and attachments received. If you have received this communication in error, please notify Matrikon Inc. by telephone at (780) 448-1010. **************************************************************************** *

1 0

RE: [PSF-Board] Spambayes
by Randy Kondor Feb. 22, 2004

Feb. 22, 2004

Folks, can you please contact me about making some enhancements to Spambayes? I can provide funding for this work. Your product is awesome!!! Regards, Randy Matrikon - We make connections! Watch the multimedia OPC Tutorial at http://www.matrikon.com/tutorial O-----------------------------------------------------------------O | Randy Kondor Phone: 780-945-4035 | | OPC Product Manager Fax: 780-448-9191 | | Matrikon Email: randy.kondor(a)matrikon.com | | 10405 Jasper Ave, Suite 1800 WEB: http://www.matrikon.com | | Edmonton, Alberta, Canada, T5J 3N4 | O-----------------------------------------------------------------O Attend Matrikon's Annual User Group May 2-5, 2004 Matrikon Valued Partners Event - MVP 2004 Driving Performance, Partners for Results. -----Original Message----- From: Tim Peters [mailto:tim.one@comcast.net] Sent: Friday, February 20, 2004 11:56 AM To: Randy Kondor Cc: psf(a)python.org Subject: RE: [PSF-Board] Spambayes [Randy Kondor] > I am interested in speaking with you about funds for Spambayes. > Please call me at 780-945-4035. Randy, the SpamBayes developers chose to give copyright to the PSF, and to release the code under the PSF license, but the PSF has no other connection with that project. For example, the PSF doesn't fund it, or manage it. You can reach the SpamBayes developers directly via: mailto:spambayes-dev@python.org All SpamBayes developers are currently unpaid volunteers, but there's no prohibition against them doingwork for pay. There's also no prohibition against using the SpamBayes code in a commercial project (and, for example, at least http://www.inboxer.com/ has done so). **************************************************************************** * READER BEWARE: Unencrypted, unsigned Internet e-mail is inherently insecure. Internet messages may be corrupted, incomplete, misdirected or may incorrectly identify the sender. Therefore, nothing in this message or attachments may be considered legally binding. THIS MESSAGE IS ONLY INTENDED FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY BE PRIVILEGED. If you are not the intended recipient or their authorized agent, you may not forward or copy this information and must delete or destroy all copies of this message and attachments received. If you have received this communication in error, please notify Matrikon Inc. by telephone at (780) 448-1010. **************************************************************************** *

1 0

RE: [spambayes-dev] Automatically generated bug reports from sb_server
by Tony Meyer Feb. 22, 2004

Feb. 22, 2004

> We could set the subject to something like > "POP3 Proxy Problem: <problem summary here>" > That *might* encourage people to change the subject, and we'll be able > to see very quickly if people aren't paying any attention to it. After reading mail today, I'm no longer hesitant - the identical subject messages have got to go. The code makes a (rudimentary) check to make sure that the user has entered in what's necessary (so we don't get lots of "...My problem is [DESCRIBE YOUR PROBLEM HERE]" messages), so I'll get it to confirm that they've done something with the subject as well. > One of the problems we have consistently is people not > providing enough information about their problem. > IIRC, one of the purposes for doing > the bug report feature was to give users a template that > would encourage them to provide the necessary info. I don't > think pointing them directly at SourceForge would do that > unless there is some way to > automatically add some template headings when opening the page. Ah, good point - I hadn't thought about that. > Is there any possibility of providing them a form to fill in > through the sb_server UI and then submitting that info directly as a > SourceForge bug instead of emailing it to the list? I think most of us > subscribe to the spambayes-bugs list anyway, so we would still get > e-mail notification. Probably not, because you have to be logged in to sourceforge to submit a bug (well, we can turn that off in the admin settings I think, but it's presumably there for a reason). I'll forget about that idea :) > Sorry, I haven't looked closely enough at the feature or the reports > received so far to provide any more input. No worries :) > I do bring up the proxy from > time to time, and I'll try to remember to look at this more. Thanks :) (Note that if you ever want to test the reporting, you can change the address it sends to in ui.html (if you have resourcepackage to regenerate ui_html.py), so it (for example) sends to you, rather than the list.) =Tony Meyer

1 0

Automatically generated bug reports from sb_server
by Tony Meyer Feb. 20, 2004

Feb. 20, 2004

Surprisingly (to me, at least), it appears that people are actually going to use the auto-generate-a-bug-report feature in sb_server. I've fixed the formatting problem, but I'm also wondering if other changes should be made to make them more effective. Should the user have to enter a subject themselves? I worry that there will be too many "Problem with POP3 Proxy" messages, and it'll be confusing to keep track of threads. (Obviously they can change the subject before posting, but there's no compulsion, or even suggestion, to do so). Would it be better to just open up a link to the sourceforge bug tracking system (they're in a web browser, after all)? Or offer both, but suggest one more strongly? Any other comments, based on the ones that have arrived so far? =Tony Meyer

2 1

a useful pre-filter for auto-training bayesian systems?
by Ryan Malayter Feb. 20, 2004

Feb. 20, 2004

I found this interesting. Using the social network as a first step, a bunch of "definite ham" and "definite spam" messages are listed. These can be used to train a Bayesian filter which then filters the rest of the unsures automatically. Very little user intervention would therefore be required for training, and it cuts in half the number of messages that must be filtered by the much-more-expensive statistical filter. http://www.arxiv.org/abs/cond-mat/0402143 One could even imagine users securely posting their email addressee's "white lists" by posting SHA-1 hashes instead of actually email addresses to some public forum. (This would have to be salted, of course). This could create a meta-social-network. They don't seem to address the issue of a spam that has a forged address from your own social network, though, which might trip up this whole social network process. Ryan Malayter Sr. Network & Database Administrator Bank Administration Institute Chicago, Illinois, USA PGP Key: http://www.malayter.com/pgp-public.txt ::::::::::::::::::::::::::::::: I am prepared to meet my Maker. Whether my Maker is prepared for the great ordeal of meeting me is another matter. -Sir Winston S. Churchill

2 1

RE: [spambayes-dev] getting rid of the "new mail" icon
by Tony Meyer Feb. 19, 2004

Feb. 19, 2004

> - Use a commercial anti-spam filter (if I can find one that > switches off the new mail icon; suggestions welcome). [...] > I am thrilled by none of these possibilities. I would have > liked to use SpamBayes, but given the amount of spam I get, > this is just not possible. You could look into InBoxer (there's a link on our 'related' page). It's the SpamBayes code (or close enough), so you'll get the same results. I believe they have done *something* to address this, although I'm not sure exactly what. In any case, since you'd be giving them money, you'd be able to ask them for features, if they don't do what you want already. =Tony Meyer

1 0