Over on the spambayes list yesterday, we were discussing a particularly good identity-theft scam spam, purporting to be from PayPal. It linked extensively to PayPal's real site, and about the only fishy lexical thing was a highly obfuscated href (full of % escapes). We don't do anything special with % escapes in URLs now. Maybe we should. The attached patch does. I don't have enough personal email saved to make for a good test, but who cares <wink>. I just took what I had, slammed into randomly into 10 even sets, and did "the usual" cross-validation business on it. All of this email is less than a week old, is all the email I've gotten since then, is atypical for me (Christmas time -> a lot less email than usual, but a spike in personal email), and runs 3:1 in favor of ham. None of that matters, though -- *whatever* you have, and however you train, the interesting question is just how it does with the patch, compared to without it. I ran my 10-fold CV with "the default" settings for Outlook. These match the current (CVS) project defaults, with the addition of [Tokenizer] replace_nonascii_chars: True record_header_absence: True I'm *not* using mine_received_headers or x-use_bigrams in these tests. befores -> afters -> <stat> tested 151 hams & 52 spams against 1359 hams & 468 spams [19 repetitions of that] false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.662 0.662 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 1 to 1 tied mean fp % went from 0.0662251655629 to 0.0662251655629 tied false negative percentages 1.923 1.923 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 1.923 1.923 tied 1.923 1.923 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fn went from 3 to 3 tied mean fn % went from 0.576923076924 to 0.576923076924 tied ham mean ham sdev 0.44 0.44 +0.00% 4.52 4.52 +0.00% 0.34 0.34 +0.00% 4.11 4.11 +0.00% 0.27 0.27 +0.00% 3.16 3.16 +0.00% 0.17 0.17 +0.00% 1.51 1.51 +0.00% 1.06 1.06 +0.00% 9.11 9.12 +0.11% 0.00 0.00 +(was 0) 0.01 0.01 +0.00% 0.78 0.78 +0.00% 8.16 8.16 +0.00% 0.42 0.43 +2.38% 5.19 5.21 +0.39% 0.01 0.01 +0.00% 0.11 0.11 +0.00% 0.07 0.07 +0.00% 0.90 0.90 +0.00% ham mean and sdev for all runs 0.36 0.36 +0.00% 4.77 4.78 +0.21% spam mean spam sdev 96.41 96.43 +0.02% 13.52 13.51 -0.07% 98.51 98.56 +0.05% 6.99 6.99 +0.00% 97.80 97.80 +0.00% 6.42 6.41 -0.16% 98.21 98.22 +0.01% 7.31 7.30 -0.14% 93.00 93.03 +0.03% 16.68 16.66 -0.12% 97.40 97.41 +0.01% 8.29 8.27 -0.24% 97.58 97.70 +0.12% 12.30 12.18 -0.98% 97.01 97.02 +0.01% 14.38 14.37 -0.07% 95.90 96.03 +0.14% 11.61 11.46 -1.29% 98.86 98.86 +0.00% 6.12 6.11 -0.16% spam mean and sdev for all runs 97.07 97.11 +0.04% 11.09 11.05 -0.36% ham/spam mean difference: 96.71 96.75 +0.04 Not much to talk about there! Pretty much indistinguishable, although the spam mean went up a tad consistently, and the spam sdev down a tad consistently. table.py's "best cost" output shows that I could have reduced the optimal cost by 1 unsure if I changed my cutoffs: filename: before after ham:spam: 1510:520 1510:520 fp total: 1 1 fp %: 0.07 0.07 fn total: 3 3 fn %: 0.58 0.58 unsure t: 39 39 unsure %: 1.92 1.92 real cost: $20.80 $20.80 best cost: $17.60 $17.40 h mean: 0.36 0.36 h sdev: 4.77 4.78 s mean: 97.07 97.11 s sdev: 11.09 11.05 mean diff: 96.71 96.75 k: 6.10 6.11 So the change would have been the tiniest of wins for me. For you? BTW, the fp here was an "end of year sale" blaring HTML ad from Gateway. That's ham to me, but there are no other msgs from Gateway in this email. It contains enough Gateway-specific lexicalisms that training on one is enough to score future ones as solid ham. The PayPal scam that started this remained a solid FN.
Tim> Over on the spambayes list yesterday, we were discussing a Tim> particularly good identity-theft scam spam, purporting to be from Tim> PayPal. It linked extensively to PayPal's real site, and about the Tim> only fishy lexical thing was a highly obfuscated href (full of % Tim> escapes). Tim> We don't do anything special with % escapes in URLs now. Maybe we Tim> should. The attached patch does. I tried a somewhat different approach (patch is attached) and got similar results (all ties at the more gross level, slight increase in spam mean and slight decrease in spam sdev, no change to ham at all (*)): stds.txt -> pickurlss.txt -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams false positive percentages 0.000 0.000 tied 0.400 0.400 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 5 times lost 0 times total unique fp went from 1 to 1 tied mean fp % went from 0.08 to 0.08 tied false negative percentages 3.333 3.333 tied 5.000 5.000 tied 7.333 7.333 tied 5.667 5.667 tied 4.000 4.000 tied won 0 times tied 5 times lost 0 times total unique fn went from 76 to 76 tied mean fn % went from 5.06666666667 to 5.06666666667 tied ham mean ham sdev 1.64 1.64 +0.00% 8.44 8.44 +0.00% 0.99 0.99 +0.00% 8.29 8.29 +0.00% 2.82 2.82 +0.00% 12.52 12.52 +0.00% 1.58 1.58 +0.00% 8.29 8.29 +0.00% 1.30 1.30 +0.00% 8.04 8.04 +0.00% ham mean and sdev for all runs 1.66 1.66 +0.00% 9.30 9.30 +0.00% spam mean spam sdev 93.80 93.82 +0.02% 19.39 19.35 -0.21% 90.56 90.58 +0.02% 24.31 24.26 -0.21% 89.24 89.27 +0.03% 27.03 27.04 +0.04% 89.27 89.27 +0.00% 25.51 25.50 -0.04% 92.72 92.74 +0.02% 21.67 21.67 +0.00% spam mean and sdev for all runs 91.12 91.14 +0.02% 23.81 23.80 -0.04% ham/spam mean difference: 89.46 89.48 +0.02 (*) Operational question: Given that my training data is somewhat small at the moment (roughly 1000-1500 each of ham and spam), would I be better off testing with fewer larger sets (e.g, 5 sets w/ 250 msgs each) or with more smaller sets (e.g, 10 sets w/ 125 msgs each)? Skip Index: spambayes/Options.py =================================================================== RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v retrieving revision 1.97 diff -c -r1.97 Options.py *** spambayes/Options.py 30 Dec 2003 16:26:33 -0000 1.97 --- spambayes/Options.py 30 Dec 2003 21:42:48 -0000 *************** *** 145,150 **** --- 145,155 ---- """(DEPRECATED) Extract day of the week tokens from the Date: header.""", BOOLEAN, RESTORE), + ("x-pick_apart_urls", "Extract clues about url structure", False, + """(EXPERIMENTAL) Note whether url contains non-standard port or + user/password elements.""", + BOOLEAN, RESTORE), + ("replace_nonascii_chars", "Replace non-ascii characters", False, """If true, replace high-bit characters (ord(c) >= 128) and control characters with question marks. This allows non-ASCII character Index: spambayes/tokenizer.py =================================================================== RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v retrieving revision 1.27 diff -c -r1.27 tokenizer.py *** spambayes/tokenizer.py 30 Dec 2003 16:26:33 -0000 1.27 --- spambayes/tokenizer.py 30 Dec 2003 21:42:48 -0000 *************** *** 13,18 **** --- 13,20 ---- import time import os import binascii + import urlparse + import urllib try: from sets import Set except ImportError: *************** *** 1014,1019 **** --- 1016,1038 ---- proto, guts = m.groups() tokens = ["proto:" + proto] pushclue = tokens.append + + if options["Tokenizer", "x-pick_apart_urls"]: + url = proto + "://" + guts + num_pcs = url.count("%") + if num_pcs: + pushclue("url:%d %%s" % num_pcs) + url = urllib.unquote(url) + scheme, netloc, path, params, query, frag = urlparse.urlparse(url) + user_pwd, host_port = urllib.splituser(netloc) + if user_pwd is not None: + pushclue("url:has user") + host, port = urllib.splitport(host_port) + if port is not None: + if scheme == "http" and port != '80': + pushclue("url:non-standard http port") + elif scheme == "https" and port != '443': + pushclue("url:non-standard https port") # Lose the trailing punctuation for casual embedding, like: # The code is at http://mystuff.org/here? Didn't resolve.
[Skip Montanaro]
I tried a somewhat different approach (patch is attached) and got similar results (all ties at the more gross level, slight increase in spam mean and slight decrease in spam sdev, no change to ham at all (*)):
3-way compare on my data: filename: before after skip ham:spam: 1510:520 1510:520 1510:520 fp total: 1 1 1 fp %: 0.07 0.07 0.07 fn total: 3 3 3 fn %: 0.58 0.58 0.58 unsure t: 39 39 39 unsure %: 1.92 1.92 1.92 real cost: $20.80 $20.80 $20.80 best cost: $17.60 $17.40 $17.80 h mean: 0.36 0.36 0.36 h sdev: 4.77 4.78 4.77 s mean: 97.07 97.11 97.08 s sdev: 11.09 11.05 11.03 mean diff: 96.71 96.75 96.72 k: 6.10 6.11 6.12 The "best cost" measure actually got marginally worse, but not significantly so. Note that this part of the patch can't be helping much: + num_pcs = url.count("%") + if num_pcs: + pushclue("url:%d %%s" % num_pcs) That is, raw counts are almost never useful -- if I have a URL in a spam that embeds 40 escapes, that does nothing to indict a URL with 39 (or 41) escapes. Pumping out log2(a_count) usually does more good. I *expect* the approach in my patch would work better, though (generating lots of correlated tokens -- there are good reasons to escape some punctuation characters in URLs, but the only good reason to escape a letter or digit is to obfuscate; let the classifier see these things, and it will learn that on its own, as appropriate, for each escape code; then a URL escaping several letters or digits will get penalized more the more heavily it employs this kind of obfuscation).
(*) Operational question: Given that my training data is somewhat small at the moment (roughly 1000-1500 each of ham and spam), would I be better off testing with fewer larger sets (e.g, 5 sets w/ 250 msgs each) or with more smaller sets (e.g, 10 sets w/ 125 msgs each)?
If you ask me <wink>, cross-validation should *always* be done with a minimum of 10 sets, regardless of how much data you have. There are many reasons for this, from statistical reliability of the grand averages at the end (they're subject to central-limit theorem constraints, and the more sets the more reliable they are, growing with the square root of the # of sets); to that it's extremely important to see run-by-run comparisons (how many runs won, lost, tied), and just about any distribution of those numbers is achievable by chance with few sets (IOW, "9 won, 1 tied, 0 lost" is very much harder to account for by chance than "4 won, 1 tied, 0 lost"; likewise "1 won, 8 tied, 1 lost" is much less likely to be produced by a significant (good or bad) change than "1 won, 3 tied, 1 lost"). Note, though, that cross-validation is modeling the performance of a train-on-everything strategy, and in random time order to boot. If that's not how you train, the results may be irrelevant to what you'll see in real life. It should be good enough to weed out really bad ideas-- and highlight really good ones --regardless, though.
Tim> Note that this part of the patch can't be helping much: Tim> + num_pcs = url.count("%") Tim> + if num_pcs: Tim> + pushclue("url:%d %%s" % num_pcs) Tim> That is, raw counts are almost never useful -- if I have a URL in a Tim> spam that embeds 40 escapes, that does nothing to indict a URL with Tim> 39 (or 41) escapes. Pumping out log2(a_count) usually does more Tim> good. I realized that before trying, but not having any raw data upon which to base things, I left it as-is. If I enable it I'll look at some results to see what tokens are actually generated and how they seem to correlate with ham and spam. One other possibility would be a sort of "Watership Down" approach: "1, 2, 3, many" (or something similar - rabbits can't count very high). The problem with log2(count) in this situation is there seems to be a practical limit to how many % signs a URL might have (maybe 50?), so something that creates buckets using division (counts // 5 ???) might do a decent job of lumping things together. I'm off work the next couple of days and have some house guests in from out of town, so I probably won't look at this much. I will try to at least build a database from my current training set using this feature and see how things shake out. (Maybe tomorrow morning before everyone's up and about.) Tim> I *expect* the approach in my patch would work better, though Tim> (generating lots of correlated tokens -- there are good reasons to Tim> escape some punctuation characters in URLs, but the only good Tim> reason to escape a letter or digit is to obfuscate; let the Tim> classifier see these things, and it will learn that on its own, as Tim> appropriate, for each escape code; then a URL escaping several Tim> letters or digits will get penalized more the more heavily it Tim> employs this kind of obfuscation). My problem with that approach is the stuff the spammers escape can be essentially random, as in the bogus URL you received. I think you might get scads of hapaxes (or at least low-count escapes). Stuff with high-counts will be legitimate (%20 and so forth). Conclusions obviously await some eyeballing of databases. >> (*) Operational question: Given that my training data is somewhat >> small at the moment (roughly 1000-1500 each of ham and spam), would I >> be better off testing with fewer larger sets (e.g, 5 sets w/ 250 msgs >> each) or with more smaller sets (e.g, 10 sets w/ 125 msgs each)? Tim> If you ask me <wink>, cross-validation should *always* be done with Tim> a minimum of 10 sets, regardless of how much data you have. There Tim> are many reasons for this, from statistical reliability of the Tim> grand averages at the end (they're subject to central-limit theorem Tim> constraints, and the more sets the more reliable they are, growing Tim> with the square root of the # of sets); Thanks, I will rebalance my training database to 10 sets and see how that goes. Tim> Note, though, that cross-validation is modeling the performance of Tim> a train-on-everything strategy, and in random time order to boot. The random time order isn't so important to me at the moment, because all the messages I'm using are recent (received within the past month or so). The "train on everything" aspect is more interesting. I find the cross-validation tests never perform as well as in real life. ;-) Tim> If that's not how you train, the results may be irrelevant to what Tim> you'll see in real life. It should be good enough to weed out Tim> really bad ideas-- and highlight really good ones --regardless, Tim> though. There's the rub. What might be really good ideas at this point will probably only result in very small changes in performance because the baseline system is currently so good. Skip
[Tim]
I *expect* the approach in my patch would work better, though (generating lots of correlated tokens -- there are good reasons to escape some punctuation characters in URLs, but the only good reason to escape a letter or digit is to obfuscate; let the classifier see these things, and it will learn that on its own, appropriate, for each escape code; then a URL escaping several letters or digits will get penalized more the more heavily it employs this kind of obfuscation).
[Skip Montanaro]
My problem with that approach is the stuff the spammers escape can be essentially random, as in the bogus URL you received. I think you might get scads of hapaxes (or at least low-count escapes). Stuff with high-counts will be legitimate (%20 and so forth).
There won't be scads of hapaxes, because the number of escape codes is finite (small, even -- only 256 make sense). I *expect* that only 62 of those will be interesting (attempts to obfuscate letters and digits), but there's no need to try to out-think that, and just sucking up every escape code without prejudice lets the classifier learn to be smarter than I am. The pre-judgment here comes from the *belief* that this is a case where generating multiple correlated clues will help more than it hurts. Especially with smaller databases, multiple clues do a lot more toward forcing a decision than a single clue can do.
Conclusions obviously await some eyeballing of databases.
Yup!
... The random time order isn't so important to me at the moment, because all the messages I'm using are recent (received within the past month or so). The "train on everything" aspect is more interesting. I find the cross-validation tests never perform as well as in real life. ;-)
I expect that's because the CV tests *do* lose time-ordering.
... There's the rub. What might be really good ideas at this point will probably only result in very small changes in performance because the baseline system is currently so good.
That's OK -- accumulating many tiny improvements is as good finding a single small improvement <wink>. That's a sure way to make ongoing progress too, and is the *usual* fate of mature statistical systems. A question remaining is whether each tiny improvement is worth the costs it incurs (in processing time, database size, and code complexity). I think this one does well on all those counts, as it only triggers in a specific context, can't add more than a few hundred tokens total to a database, and the code is simple.
Tim> Note that this part of the patch can't be helping much: Tim> + num_pcs = url.count("%") Tim> + if num_pcs: Tim> + pushclue("url:%d %%s" % num_pcs) Tim> That is, raw counts are almost never useful -- if I have a URL in a Tim> spam that embeds 40 escapes, that does nothing to indict a URL with Tim> 39 (or 41) escapes. Pumping out log2(a_count) usually does more Tim> good. <aside type="slight"> "url:has user" seems to be fairly spammy for me: % spamcounts -r -d ~/tmp/hammie.db '^url:has user' db: /Users/skip/tmp/hammie.db token,nspam,nham,spam prob url:has user,42,4,0.91016660508 </aside> Okay, here are the raw number of URL percents as present in my current ham/spam database: npcs nspam nham 1 21 46 2 4 1 3 2 2 4 1 2 5 0 1 6 2 2 7 1 1 8 0 2 14 2 0 15 0 1 16 1 0 18 1 0 23 1 0 24 1 0 28 1 0 30 1 0 38 2 0 40 1 0 42 1 0 74 1 0 75 1 0 84 1 0 97 1 0 103 1 0 109 1 0 191 1 0 I redid my patch to generate tokens like so: pushclue("url:%%%d" % int(log2(num_pcs))) Converting the first column to int(log(n,2)) then rebuilding the database gives: log(npcs) nspam nham 0 21 46 1 6 3 2 4 2 3 2 2 4 5 0 5 3 0 6 2 0 7 1 0 The new cv test results are essentially the same (I still have just five sets): stds.txt -> pickurlss.txt -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams -> <stat> tested 250 hams & 300 spams against 1000 hams & 1200 spams false positive percentages 0.000 0.000 tied 0.400 0.400 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 5 times lost 0 times total unique fp went from 1 to 1 tied mean fp % went from 0.08 to 0.08 tied false negative percentages 3.333 3.333 tied 5.000 5.000 tied 7.333 7.333 tied 5.667 5.667 tied 4.000 4.000 tied won 0 times tied 5 times lost 0 times total unique fn went from 76 to 76 tied mean fn % went from 5.06666666667 to 5.06666666667 tied ham mean ham sdev 1.64 1.64 +0.00% 8.44 8.45 +0.12% 0.99 0.99 +0.00% 8.29 8.29 +0.00% 2.82 2.82 +0.00% 12.52 12.52 +0.00% 1.58 1.58 +0.00% 8.29 8.29 +0.00% 1.30 1.30 +0.00% 8.04 8.04 +0.00% ham mean and sdev for all runs 1.66 1.66 +0.00% 9.30 9.30 +0.00% spam mean spam sdev 93.80 93.83 +0.03% 19.39 19.31 -0.41% 90.56 90.59 +0.03% 24.31 24.26 -0.21% 89.24 89.28 +0.04% 27.03 27.04 +0.04% 89.27 89.27 +0.00% 25.51 25.50 -0.04% 92.72 92.74 +0.02% 21.67 21.67 +0.00% spam mean and sdev for all runs 91.12 91.14 +0.02% 23.81 23.79 -0.08% ham/spam mean difference: 89.46 89.48 +0.02 Skip
Happy New Year everyone... As Tim predicted, mixing his url cracking ideas with mine leads to better performance than either of our ideas in isolation. Using the attached patch, I get this summary output for a 10x10 timcv run: stds.txt -> pickurlss.txt -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 1.667 1.667 tied 0.833 0.833 tied 0.833 0.833 tied 0.000 0.000 tied 0.000 0.000 tied 0.833 0.833 tied won 0 times tied 10 times lost 0 times total unique fp went from 5 to 5 tied mean fp % went from 0.416666666667 to 0.416666666667 tied false negative percentages 7.874 7.874 tied 6.299 6.299 tied 9.449 9.449 tied 9.449 9.449 tied 10.236 10.236 tied 5.512 5.512 tied 7.087 6.299 won -11.12% 5.556 5.556 tied 7.937 7.937 tied 8.661 8.661 tied won 1 times tied 9 times lost 0 times total unique fn went from 99 to 98 won -1.01% mean fn % went from 7.80589926259 to 7.72715910511 won -1.01% ham mean ham sdev 2.11 2.12 +0.47% 12.36 12.36 +0.00% 3.28 3.33 +1.52% 14.07 14.13 +0.43% 1.11 1.13 +1.80% 6.75 6.86 +1.63% 1.13 1.12 -0.88% 5.90 5.86 -0.68% 3.44 3.43 -0.29% 14.07 14.06 -0.07% 3.66 3.65 -0.27% 15.31 15.30 -0.07% 3.68 3.67 -0.27% 13.65 13.62 -0.22% 1.10 1.10 +0.00% 6.93 6.93 +0.00% 1.70 1.78 +4.71% 8.80 9.02 +2.50% 3.49 3.49 +0.00% 14.57 14.58 +0.07% ham mean and sdev for all runs 2.47 2.48 +0.40% 11.83 11.85 +0.17% spam mean spam sdev 84.79 84.96 +0.20% 29.71 29.56 -0.50% 88.72 88.85 +0.15% 26.91 26.91 +0.00% 83.53 83.99 +0.55% 30.40 30.26 -0.46% 85.69 85.97 +0.33% 29.57 29.60 +0.10% 84.47 84.59 +0.14% 30.42 30.45 +0.10% 89.08 89.25 +0.19% 24.73 24.56 -0.69% 87.08 87.73 +0.75% 27.80 27.05 -2.70% 88.44 88.48 +0.05% 25.70 25.67 -0.12% 87.20 87.23 +0.03% 28.53 28.54 +0.04% 86.46 86.47 +0.01% 27.85 27.88 +0.11% spam mean and sdev for all runs 86.54 86.75 +0.24% 28.28 28.17 -0.39% ham/spam mean difference: 84.07 84.27 +0.20 I also ran with bigrams enabled. That helped more: stds.txt -> pickbis.txt -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 126 spams against 1080 hams & 1142 spams -> <stat> tested 120 hams & 127 spams against 1080 hams & 1141 spams false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 1.667 1.667 tied 0.833 0.833 tied 0.833 0.833 tied 0.000 0.833 lost +(was 0) 0.000 0.000 tied 0.833 0.833 tied won 0 times tied 9 times lost 1 times total unique fp went from 5 to 6 lost +20.00% mean fp % went from 0.416666666667 to 0.5 lost +20.00% false negative percentages 7.874 6.299 won -20.00% 6.299 4.724 won -25.00% 9.449 6.299 won -33.34% 9.449 5.512 won -41.67% 10.236 4.724 won -53.85% 5.512 1.575 won -71.43% 7.087 5.512 won -22.22% 5.556 5.556 tied 7.937 7.937 tied 8.661 2.362 won -72.73% won 8 times tied 2 times lost 0 times total unique fn went from 99 to 64 won -35.35% mean fn % went from 7.80589926259 to 5.04999375078 won -35.31% ham mean ham sdev 2.11 1.61 -23.70% 12.36 10.88 -11.97% 3.28 2.85 -13.11% 14.07 12.69 -9.81% 1.11 1.05 -5.41% 6.75 6.13 -9.19% 1.13 1.00 -11.50% 5.90 4.72 -20.00% 3.44 3.19 -7.27% 14.07 14.75 +4.83% 3.66 3.45 -5.74% 15.31 15.27 -0.26% 3.68 2.67 -27.45% 13.65 11.70 -14.29% 1.10 1.85 +68.18% 6.93 10.11 +45.89% 1.70 1.93 +13.53% 8.80 9.23 +4.89% 3.49 3.31 -5.16% 14.57 14.97 +2.75% ham mean and sdev for all runs 2.47 2.29 -7.29% 11.83 11.60 -1.94% spam mean spam sdev 84.79 86.82 +2.39% 29.71 27.17 -8.55% 88.72 90.26 +1.74% 26.91 24.30 -9.70% 83.53 87.45 +4.69% 30.40 26.76 -11.97% 85.69 88.25 +2.99% 29.57 27.35 -7.51% 84.47 88.02 +4.20% 30.42 25.64 -15.71% 89.08 92.22 +3.52% 24.73 21.06 -14.84% 87.08 91.45 +5.02% 27.80 23.48 -15.54% 88.44 89.02 +0.66% 25.70 26.08 +1.48% 87.20 87.78 +0.67% 28.53 28.58 +0.18% 86.46 90.65 +4.85% 27.85 23.02 -17.34% spam mean and sdev for all runs 86.54 89.19 +3.06% 28.28 25.50 -9.83% ham/spam mean difference: 84.07 86.90 +2.83 Skip Index: spambayes/Options.py =================================================================== RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v retrieving revision 1.97 diff -c -r1.97 Options.py *** spambayes/Options.py 30 Dec 2003 16:26:33 -0000 1.97 --- spambayes/Options.py 2 Jan 2004 13:57:56 -0000 *************** *** 145,150 **** --- 145,155 ---- """(DEPRECATED) Extract day of the week tokens from the Date: header.""", BOOLEAN, RESTORE), + ("x-pick_apart_urls", "Extract clues about url structure", False, + """(EXPERIMENTAL) Note whether url contains non-standard port or + user/password elements.""", + BOOLEAN, RESTORE), + ("replace_nonascii_chars", "Replace non-ascii characters", False, """If true, replace high-bit characters (ord(c) >= 128) and control characters with question marks. This allows non-ASCII character Index: spambayes/tokenizer.py =================================================================== RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v retrieving revision 1.27 diff -c -r1.27 tokenizer.py *** spambayes/tokenizer.py 30 Dec 2003 16:26:33 -0000 1.27 --- spambayes/tokenizer.py 2 Jan 2004 13:57:56 -0000 *************** *** 13,18 **** --- 13,20 ---- import time import os import binascii + import urlparse + import urllib try: from sets import Set except ImportError: *************** *** 1012,1025 **** def tokenize(self, m): proto, guts = m.groups() tokens = ["proto:" + proto] pushclue = tokens.append # Lose the trailing punctuation for casual embedding, like: # The code is at http://mystuff.org/here? Didn't resolve. # or # I found it at http://mystuff.org/there/. Thanks! - assert guts while guts and guts[-1] in '.:?!/': guts = guts[:-1] for piece in guts.split('/'): --- 1014,1073 ---- def tokenize(self, m): proto, guts = m.groups() + assert guts tokens = ["proto:" + proto] pushclue = tokens.append + if options["Tokenizer", "x-pick_apart_urls"]: + url = proto + "://" + guts + + escapes = re.findall(r'%..', guts) + # roughly how many %nn escapes are there? + if escapes: + pushclue("url:%%%d" % int(log2(len(escapes)))) + # %nn escapes are usually intentional obfuscation. Generate a + # lot of correlated tokens if the URL contains a lot of them. + # The classifier will learn which specific ones are and aren't + # spammy. + tokens.extend(["url:" + escape for escape in escapes]) + + # now remove any obfuscation and probe around a bit + url = urllib.unquote(url) + scheme, netloc, path, params, query, frag = urlparse.urlparse(url) + + # one common technique in bogus "please (re-)authorize yourself" + # scams is to make it appear as if you're visiting a valid + # payment-oriented site like PayPal, CitiBank or eBay, when you + # actually aren't. The company's web server appears as the + # beginning of an often long username element in the URL such as + # http://www.paypal.com%65%43%99%35@10.0.1.1/iwantyourccinfo + # generally with an innocuous-looking fragment of text or a + # valid URL as the highlighted link. Usernames should rarely + # appear in URLs (perhaps in a local bookmark you established), + # and never in a URL you receive from an unsolicited email or + # another website. + user_pwd, host_port = urllib.splituser(netloc) + if user_pwd is not None: + pushclue("url:has user") + + host, port = urllib.splitport(host_port) + # web servers listening on non-standard ports are suspicious ... + if port is not None: + if (scheme == "http" and port != '80' or + scheme == "https" and port != '443'): + pushclue("url:non-standard %s port" % scheme) + + # ... as are web servers associated with raw ip addresses + if re.match("(\d+\.?){4,4}$", host) is not None: + pushclue("url:ip addr") + + # make sure we later tokenize the unobfuscated url bits + proto, guts = url.split("://", 1) + # Lose the trailing punctuation for casual embedding, like: # The code is at http://mystuff.org/here? Didn't resolve. # or # I found it at http://mystuff.org/there/. Thanks! while guts and guts[-1] in '.:?!/': guts = guts[:-1] for piece in guts.split('/'):
Here are my current results with Skip's latest patch; "url" is the same as "base" except with the addition of x-pick_apart_urls: True bases -> urls -> <stat> tested 342 hams & 94 spams against 3078 hams & 846 spams <19 repetitions deleted> false positive percentages 0.292 0.292 tied 0.000 0.000 tied 0.000 0.000 tied 0.292 0.292 tied 0.000 0.000 tied 0.000 0.000 tied 0.292 0.292 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 3 to 3 tied mean fp % went from 0.0877192982457 to 0.0877192982457 tied false negative percentages 2.128 2.128 tied 0.000 0.000 tied 0.000 0.000 tied 1.064 1.064 tied 2.128 2.128 tied 2.128 2.128 tied 2.128 2.128 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fn went from 9 to 9 tied mean fn % went from 0.957446808511 to 0.957446808511 tied ham mean ham sdev 0.51 0.51 +0.00% 5.96 5.96 +0.00% 0.12 0.12 +0.00% 1.08 1.09 +0.93% 0.44 0.44 +0.00% 4.55 4.55 +0.00% 0.39 0.39 +0.00% 5.59 5.59 +0.00% 0.49 0.49 +0.00% 4.58 4.60 +0.44% 0.84 0.85 +1.19% 6.12 6.18 +0.98% 0.47 0.47 +0.00% 5.60 5.60 +0.00% 0.34 0.34 +0.00% 3.15 3.15 +0.00% 0.20 0.20 +0.00% 2.08 2.08 +0.00% 0.08 0.08 +0.00% 0.88 0.89 +1.14% ham mean and sdev for all runs 0.39 0.39 +0.00% 4.40 4.41 +0.23% spam mean spam sdev 94.15 94.16 +0.01% 17.84 17.83 -0.06% 98.85 98.87 +0.02% 4.99 4.94 -1.00% 98.07 98.34 +0.28% 6.49 5.99 -7.70% 96.98 96.99 +0.01% 13.46 13.49 +0.22% 96.21 96.25 +0.04% 15.89 15.83 -0.38% 94.07 94.07 +0.00% 17.29 17.29 +0.00% 95.61 95.65 +0.04% 16.66 16.65 -0.06% 96.62 96.66 +0.04% 11.43 11.16 -2.36% 99.25 99.27 +0.02% 2.55 2.55 +0.00% 97.43 97.44 +0.01% 9.85 9.82 -0.30% spam mean and sdev for all runs 96.72 96.77 +0.05% 12.88 12.82 -0.47% ham/spam mean difference: 96.33 96.38 +0.05 filename: base url ham:spam: 3420:940 3420:940 fp total: 3 3 fp %: 0.09 0.09 fn total: 9 9 fn %: 0.96 0.96 unsure t: 80 79 unsure %: 1.83 1.81 real cost: $55.00 $54.80 best cost: $43.80 $43.00 h mean: 0.39 0.39 h sdev: 4.40 4.41 s mean: 96.72 96.77 s sdev: 12.88 12.82 mean diff: 96.33 96.38 k: 5.57 5.59 It's not hurting <wink>. Skip, why don't you check this in, so we can try to make testing easier for others? I'm fine with making it the default behavior, provided we get decent test results from more people. [& Skip tests bigrams]
... false negative percentages 7.874 6.299 won -20.00% 6.299 4.724 won -25.00% 9.449 6.299 won -33.34% 9.449 5.512 won -41.67% 10.236 4.724 won -53.85% 5.512 1.575 won -71.43% 7.087 5.512 won -22.22% 5.556 5.556 tied 7.937 7.937 tied 8.661 2.362 won -72.73%
won 8 times tied 2 times lost 0 times
That's a clear significant win for you , eh? I'm a little baffled by my results. In real life day-to-day use, bigrams are doing great for me, under mistake-and-unsure training + artificially forcing balance by "random eyeball" selection. But CV testing shows a very small improvement (under randomized TOE): bayes\testtools>\python23\python cmp.py bases bis bases -> bis -> <stat> tested 342 hams & 94 spams against 3078 hams & 846 spams <19 repetitions deleted> false positive percentages 0.292 0.292 tied 0.000 0.000 tied 0.000 0.000 tied 0.292 0.292 tied 0.000 0.000 tied 0.000 0.000 tied 0.292 0.000 won -100.00% 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 1 times tied 9 times lost 0 times total unique fp went from 3 to 2 won -33.33% mean fp % went from 0.0877192982457 to 0.0584795321638 won -33.33% false negative percentages 2.128 2.128 tied 0.000 0.000 tied 0.000 0.000 tied 1.064 1.064 tied 2.128 2.128 tied 2.128 1.064 won -50.00% 2.128 2.128 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 1 times tied 9 times lost 0 times total unique fn went from 9 to 8 won -11.11% mean fn % went from 0.957446808511 to 0.851063829787 won -11.11% ham mean ham sdev 0.51 0.48 -5.88% 5.96 5.96 +0.00% 0.12 0.20 +66.67% 1.08 1.76 +62.96% 0.44 0.49 +11.36% 4.55 4.49 -1.32% 0.39 0.43 +10.26% 5.59 5.79 +3.58% 0.49 0.57 +16.33% 4.58 5.28 +15.28% 0.84 0.75 -10.71% 6.12 5.54 -9.48% 0.47 0.31 -34.04% 5.60 3.59 -35.89% 0.34 0.52 +52.94% 3.15 4.77 +51.43% 0.20 0.21 +5.00% 2.08 2.26 +8.65% 0.08 0.04 -50.00% 0.88 0.52 -40.91% ham mean and sdev for all runs 0.39 0.40 +2.56% 4.40 4.38 -0.45% spam mean spam sdev 94.15 93.92 -0.24% 17.84 18.18 +1.91% 98.85 98.04 -0.82% 4.99 8.00 +60.32% 98.07 97.66 -0.42% 6.49 9.05 +39.45% 96.98 96.98 +0.00% 13.46 13.56 +0.74% 96.21 95.06 -1.20% 15.89 17.58 +10.64% 94.07 94.06 -0.01% 17.29 17.26 -0.17% 95.61 95.65 +0.04% 16.66 16.39 -1.62% 96.62 96.85 +0.24% 11.43 10.39 -9.10% 99.25 98.74 -0.51% 2.55 7.78 +205.10% 97.43 96.83 -0.62% 9.85 11.72 +18.98% spam mean and sdev for all runs 96.72 96.38 -0.35% 12.88 13.66 +6.06% ham/spam mean difference: 96.33 95.98 -0.35 filename: base bi ham:spam: 3420:940 3420:940 fp total: 3 2 fp %: 0.09 0.06 fn total: 9 8 fn %: 0.96 0.85 unsure t: 80 84 unsure %: 1.83 1.93 real cost: $55.00 $44.80 best cost: $43.80 $39.40 h mean: 0.39 0.40 h sdev: 4.40 4.38 s mean: 96.72 96.38 s sdev: 12.88 13.66 mean diff: 96.33 95.98 k: 5.57 5.32
Tim> [& Skip tests bigrams] >> ... >> false negative percentages >> 7.874 6.299 won -20.00% >> 6.299 4.724 won -25.00% >> 9.449 6.299 won -33.34% >> 9.449 5.512 won -41.67% >> 10.236 4.724 won -53.85% >> 5.512 1.575 won -71.43% >> 7.087 5.512 won -22.22% >> 5.556 5.556 tied >> 7.937 7.937 tied >> 8.661 2.362 won -72.73% >> >> won 8 times >> tied 2 times >> lost 0 times Tim> That's a clear significant win for you , eh? Yeah, but note that my fn & unsure percentages (at least in test scenarios) are pretty high. Given that, it's not all that surprising that I get a bigger boost from bigrams than you do. I have yet to figure out why mine are so bad. I haven't found many misclassified messages (down in the onesies and twosies range with over 1000 each of ham and spam). I really need to implement that secondary database that maps clues to messages. Skip
Here are my test results against 2021 hams and 1942 spams spread evenly across 10 sets. The test set comes from a complete capture of my e-mail stream from a couple of months ago, plus a few more recent mails that were still lying around in my mail folders and recent training data. ============================================================ Comparison of pick_apart_urls with mine_received_headers set to False: false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 0 to 0 tied mean fp % went from 0.0 to 0.0 tied false negative percentages 1.026 1.026 tied 2.051 1.538 won -25.01% 2.577 1.546 won -40.01% 5.155 4.124 won -20.00% 2.062 1.546 won -25.02% 4.639 4.124 won -11.10% 3.608 3.093 won -14.27% 6.186 4.124 won -33.33% 3.093 3.093 tied 3.608 2.577 won -28.58% won 8 times tied 2 times lost 0 times total unique fn went from 66 to 52 won -21.21% mean fn % went from 3.40047581285 to 2.67909066878 won -21.21% ham mean ham sdev 0.34 0.34 +0.00% 4.72 4.78 +1.27% 0.03 0.03 +0.00% 0.38 0.38 +0.00% 0.17 0.19 +11.76% 1.79 1.82 +1.68% 0.08 0.08 +0.00% 0.73 0.75 +2.74% 0.06 0.06 +0.00% 0.64 0.65 +1.56% 0.10 0.10 +0.00% 1.45 1.47 +1.38% 0.02 0.02 +0.00% 0.32 0.32 +0.00% 0.28 0.28 +0.00% 3.93 3.93 +0.00% 0.05 0.05 +0.00% 0.75 0.75 +0.00% 0.00 0.00 +(was 0) 0.00 0.00 +(was 0) ham mean and sdev for all runs 0.11 0.12 +9.09% 2.12 2.14 +0.94% spam mean spam sdev 93.87 94.76 +0.95% 16.36 15.16 -7.33% 95.16 95.67 +0.54% 16.65 15.28 -8.23% 93.93 94.92 +1.05% 18.64 16.68 -10.52% 90.62 91.60 +1.08% 24.57 22.95 -6.59% 93.95 94.55 +0.64% 18.31 17.23 -5.90% 91.06 92.13 +1.18% 22.59 21.43 -5.14% 91.77 92.38 +0.66% 21.80 21.14 -3.03% 91.32 92.28 +1.05% 24.35 22.21 -8.79% 92.67 93.66 +1.07% 20.41 19.35 -5.19% 92.45 93.44 +1.07% 21.54 20.09 -6.73% spam mean and sdev for all runs 92.68 93.54 +0.93% 20.76 19.39 -6.60% ham/spam mean difference: 92.57 93.42 +0.85 ============================================================ Comparison of pick_apart_urls with mine_received_headers set to True: false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 0 to 0 tied mean fp % went from 0.0 to 0.0 tied false negative percentages 1.026 0.513 won -50.00% 1.026 0.000 won -100.00% 0.515 0.000 won -100.00% 3.608 2.577 won -28.58% 1.546 1.546 tied 2.577 2.577 tied 3.093 3.093 tied 3.608 2.062 won -42.85% 1.546 1.031 won -33.31% 2.062 1.546 won -25.02% won 7 times tied 3 times lost 0 times total unique fn went from 40 to 29 won -27.50% mean fn % went from 2.06079830822 to 1.49458102035 won -27.48% ham mean ham sdev 0.33 0.34 +3.03% 4.72 4.78 +1.27% 0.00 0.00 +(was 0) 0.03 0.03 +0.00% 0.11 0.12 +9.09% 1.42 1.43 +0.70% 0.00 0.00 +(was 0) 0.03 0.03 +0.00% 0.00 0.00 +(was 0) 0.04 0.04 +0.00% 0.02 0.02 +0.00% 0.21 0.22 +4.76% 0.00 0.00 +(was 0) 0.00 0.00 +(was 0) 0.37 0.37 +0.00% 5.20 5.20 +0.00% 0.00 0.00 +(was 0) 0.00 0.00 +(was 0) 0.00 0.00 +(was 0) 0.00 0.00 +(was 0) ham mean and sdev for all runs 0.08 0.08 +0.00% 2.27 2.28 +0.44% spam mean spam sdev 95.88 96.44 +0.58% 13.46 12.43 -7.65% 96.85 97.24 +0.40% 12.37 10.69 -13.58% 96.07 96.71 +0.67% 13.65 12.16 -10.92% 93.32 94.08 +0.81% 20.36 18.68 -8.25% 95.54 95.80 +0.27% 15.56 14.91 -4.18% 94.20 94.72 +0.55% 18.30 17.73 -3.11% 93.52 93.83 +0.33% 19.72 19.28 -2.23% 93.51 94.31 +0.86% 19.99 18.23 -8.80% 94.99 95.46 +0.49% 17.11 16.41 -4.09% 94.95 95.42 +0.49% 17.05 16.01 -6.10% spam mean and sdev for all runs 94.88 95.40 +0.55% 17.02 15.95 -6.29% ham/spam mean difference: 94.80 95.32 +0.52 ============================================================ And finally, here is the table.py comparison of all four option combinations: filename: base pick_apart_urls received+urls mine_received ham:spam: 2021:1942 2021:1942 2021:1942 2021:1942 fp total: 0 0 0 0 fp %: 0.00 0.00 0.00 0.00 fn total: 66 52 40 29 fn %: 3.40 2.68 2.06 1.49 unsure t: 200 187 159 155 unsure %: 5.05 4.72 4.01 3.91 real cost: $106.00 $89.40 $71.80 $60.00 best cost: $53.60 $50.00 $41.60 $39.60 h mean: 0.11 0.12 0.08 0.08 h sdev: 2.12 2.14 2.27 2.28 s mean: 92.68 93.54 94.88 95.40 s sdev: 20.76 19.39 17.02 15.95 mean diff: 92.57 93.42 94.80 95.32 k: 4.05 4.34 4.91 5.23 -- Kenny Pitt
[Kenny Pitt, tests x-pick_apart_urls and mine_received_headers] Remarkable results, Kenny! I want your email mix <wink>. No effect on FP, major reductions in FN and Unsure rates, and
won 7 times tied 3 times lost 0 times
was your weakest outcome. These options are as close to a pure win on your email as we've seen since The Early Days. Anyone else? Skip and Kenny both reported (surprisingly, to me) strong benefits from these gimmicks, and nobody yet has reported anything bad from them. If nobody does, there's no reason not to make them default behaviors (and to promote pick_apart_urls up from experimental status).
Tim Peters wrote:
[Kenny Pitt, tests x-pick_apart_urls and mine_received_headers]
Remarkable results, Kenny! I want your email mix <wink>. No effect on FP, major reductions in FN and Unsure rates, and
won 7 times tied 3 times lost 0 times
was your weakest outcome. These options are as close to a pure win on your email as we've seen since The Early Days.
I have the distinct advantage that 99% of my ham is either messages from a small set of mailing lists such as spambayes-dev, or company mail sent to the same list of 20 or so people. My training data quickly develops the "defacto whitelist" effect. Hopefully that doesn't skew my testing results too much, but I suppose it's the variety of e-mail mixes that makes testing useful anyway. -- Kenny Pitt
[Kenny Pitt]
I have the distinct advantage that 99% of my ham is either messages from a small set of mailing lists such as spambayes-dev, or company mail sent to the same list of 20 or so people. My training data quickly develops the "defacto whitelist" effect. Hopefully that doesn't skew my testing results too much, but I suppose it's the variety of e-mail mixes that makes testing useful anyway.
Indeed it is -- email mix varies *a lot* across people, and this is one of the only projects I know of that tests with multiple real-life personal corpora. Your mix may not be like mine, but I bet it's like thousands of others (as is mine, but a different bunch of thousands) -- all testing is appreciated here, and the greater the variety the more sure we can be that changes are truly winners.
participants (3)
-
Kenny Pitt -
Skip Montanaro -
Tim Peters