RE: [spambayes-dev] A URL experiment
My results (this is with a chuck of my most recent mail, with timcv.py -n10). Tim's patch: bases.txt -> nntims.txt -> <stat> tested 357 hams & 395 spams against 3311 hams & 3704 spams -> <stat> tested 397 hams & 384 spams against 3271 hams & 3715 spams -> <stat> tested 385 hams & 433 spams against 3283 hams & 3666 spams -> <stat> tested 407 hams & 397 spams against 3261 hams & 3702 spams -> <stat> tested 350 hams & 412 spams against 3318 hams & 3687 spams -> <stat> tested 338 hams & 405 spams against 3330 hams & 3694 spams -> <stat> tested 359 hams & 416 spams against 3309 hams & 3683 spams -> <stat> tested 358 hams & 405 spams against 3310 hams & 3694 spams -> <stat> tested 348 hams & 411 spams against 3320 hams & 3688 spams -> <stat> tested 369 hams & 441 spams against 3299 hams & 3658 spams -> <stat> tested 357 hams & 395 spams against 3311 hams & 3704 spams -> <stat> tested 397 hams & 384 spams against 3271 hams & 3715 spams -> <stat> tested 385 hams & 433 spams against 3283 hams & 3666 spams -> <stat> tested 407 hams & 397 spams against 3261 hams & 3702 spams -> <stat> tested 350 hams & 412 spams against 3318 hams & 3687 spams -> <stat> tested 338 hams & 405 spams against 3330 hams & 3694 spams -> <stat> tested 359 hams & 416 spams against 3309 hams & 3683 spams -> <stat> tested 358 hams & 405 spams against 3310 hams & 3694 spams -> <stat> tested 348 hams & 411 spams against 3320 hams & 3688 spams -> <stat> tested 369 hams & 441 spams against 3299 hams & 3658 spams false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.246 0.246 tied 0.000 0.000 tied 0.000 0.000 tied 0.557 0.557 tied 0.559 0.559 tied 0.287 0.287 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 6 to 6 tied mean fp % went from 0.164881884948 to 0.164881884948 tied false negative percentages 0.253 0.253 tied 0.781 0.781 tied 0.462 0.462 tied 0.756 0.756 tied 0.243 0.243 tied 0.247 0.247 tied 0.240 0.240 tied 0.494 0.494 tied 0.973 0.973 tied 0.454 0.454 tied won 0 times tied 10 times lost 0 times total unique fn went from 20 to 20 tied mean fn % went from 0.490257037938 to 0.490257037938 tied ham mean ham sdev 1.18 1.17 -0.85% 7.76 7.67 -1.16% 0.99 0.99 +0.00% 6.64 6.64 +0.00% 0.84 0.85 +1.19% 6.14 6.14 +0.00% 1.99 2.10 +5.53% 9.46 9.73 +2.85% 0.49 0.49 +0.00% 3.59 3.57 -0.56% 0.85 0.87 +2.35% 5.45 5.46 +0.18% 1.16 1.16 +0.00% 9.30 9.29 -0.11% 1.20 1.30 +8.33% 8.13 8.66 +6.52% 1.55 1.55 +0.00% 8.05 8.05 +0.00% 0.47 0.47 +0.00% 3.22 3.15 -2.17% ham mean and sdev for all runs 1.08 1.10 +1.85% 7.13 7.21 +1.12% spam mean spam sdev 98.75 98.75 +0.00% 8.72 8.72 +0.00% 97.67 97.69 +0.02% 11.26 11.24 -0.18% 98.08 98.14 +0.06% 10.12 9.97 -1.48% 98.16 98.16 +0.00% 10.19 10.20 +0.10% 98.35 98.41 +0.06% 8.77 8.69 -0.91% 98.45 98.47 +0.02% 8.97 8.86 -1.23% 98.35 98.41 +0.06% 9.73 9.69 -0.41% 98.25 98.36 +0.11% 9.16 8.96 -2.18% 97.93 97.97 +0.04% 11.99 11.98 -0.08% 98.92 98.93 +0.01% 7.62 7.62 +0.00% spam mean and sdev for all runs 98.30 98.34 +0.04% 9.72 9.66 -0.62% ham/spam mean difference: 97.22 97.24 +0.02 Skip's patch: bases.txt -> pickskips.txt -> <stat> tested 357 hams & 395 spams against 3311 hams & 3704 spams -> <stat> tested 397 hams & 384 spams against 3271 hams & 3715 spams -> <stat> tested 385 hams & 433 spams against 3283 hams & 3666 spams -> <stat> tested 407 hams & 397 spams against 3261 hams & 3702 spams -> <stat> tested 350 hams & 412 spams against 3318 hams & 3687 spams -> <stat> tested 338 hams & 405 spams against 3330 hams & 3694 spams -> <stat> tested 359 hams & 416 spams against 3309 hams & 3683 spams -> <stat> tested 358 hams & 405 spams against 3310 hams & 3694 spams -> <stat> tested 348 hams & 411 spams against 3320 hams & 3688 spams -> <stat> tested 369 hams & 441 spams against 3299 hams & 3658 spams -> <stat> tested 357 hams & 395 spams against 3311 hams & 3704 spams -> <stat> tested 397 hams & 384 spams against 3271 hams & 3715 spams -> <stat> tested 385 hams & 433 spams against 3283 hams & 3666 spams -> <stat> tested 407 hams & 397 spams against 3261 hams & 3702 spams -> <stat> tested 350 hams & 412 spams against 3318 hams & 3687 spams -> <stat> tested 338 hams & 405 spams against 3330 hams & 3694 spams -> <stat> tested 359 hams & 416 spams against 3309 hams & 3683 spams -> <stat> tested 358 hams & 405 spams against 3310 hams & 3694 spams -> <stat> tested 348 hams & 411 spams against 3320 hams & 3688 spams -> <stat> tested 369 hams & 441 spams against 3299 hams & 3658 spams false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.246 0.246 tied 0.000 0.000 tied 0.000 0.000 tied 0.557 0.557 tied 0.559 0.559 tied 0.287 0.287 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 6 to 6 tied mean fp % went from 0.164881884948 to 0.164881884948 tied false negative percentages 0.253 0.253 tied 0.781 0.781 tied 0.462 0.462 tied 0.756 0.756 tied 0.243 0.243 tied 0.247 0.247 tied 0.240 0.240 tied 0.494 0.494 tied 0.973 0.973 tied 0.454 0.454 tied won 0 times tied 10 times lost 0 times total unique fn went from 20 to 20 tied mean fn % went from 0.490257037938 to 0.490257037938 tied ham mean ham sdev 1.18 1.18 +0.00% 7.76 7.76 +0.00% 0.99 0.99 +0.00% 6.64 6.64 +0.00% 0.84 0.84 +0.00% 6.14 6.14 +0.00% 1.99 1.99 +0.00% 9.46 9.46 +0.00% 0.49 0.50 +2.04% 3.59 3.60 +0.28% 0.85 0.87 +2.35% 5.45 5.55 +1.83% 1.16 1.16 +0.00% 9.30 9.30 +0.00% 1.20 1.21 +0.83% 8.13 8.14 +0.12% 1.55 1.55 +0.00% 8.05 8.06 +0.12% 0.47 0.47 +0.00% 3.22 3.22 +0.00% ham mean and sdev for all runs 1.08 1.08 +0.00% 7.13 7.14 +0.14% spam mean spam sdev 98.75 98.78 +0.03% 8.72 8.56 -1.83% 97.67 97.70 +0.03% 11.26 11.25 -0.09% 98.08 98.08 +0.00% 10.12 10.12 +0.00% 98.16 98.17 +0.01% 10.19 10.15 -0.39% 98.35 98.38 +0.03% 8.77 8.73 -0.46% 98.45 98.46 +0.01% 8.97 8.97 +0.00% 98.35 98.38 +0.03% 9.73 9.68 -0.51% 98.25 98.29 +0.04% 9.16 9.05 -1.20% 97.93 97.95 +0.02% 11.99 11.98 -0.08% 98.92 98.93 +0.01% 7.62 7.62 +0.00% spam mean and sdev for all runs 98.30 98.32 +0.02% 9.72 9.68 -0.41% ham/spam mean difference: 97.22 97.24 +0.02 3-way compare: filename: bases nntims pickskips ham:spam: 3668:4099 3668:4099 3668:4099 fp total: 6 6 6 fp %: 0.16 0.16 0.16 fn total: 20 20 20 fn %: 0.49 0.49 0.49 unsure t: 178 173 175 unsure %: 2.29 2.23 2.25 real cost: $115.60 $114.60 $115.00 best cost: $93.00 $91.20 $92.40 h mean: 1.08 1.10 1.08 h sdev: 7.13 7.21 7.14 s mean: 98.30 98.34 98.32 s sdev: 9.72 9.66 9.68 mean diff: 97.22 97.24 97.24 k: 5.77 5.76 5.78 Rather like Tim's results, really, at least to my ignorant eyes. =Tony Meyer
[Tony Meyer, tries the patches] Thanks, Tony!
... Rather like Tim's results, really, at least to my ignorant eyes.
The results are both as weakly positive as things get, but at least neither patch is doing any harm. As before, I'd rather see Skip try to deal with % escapes the way my patch did -- that's a common obfuscation trick, and I bet it accounts for the small reduction in Unsures you saw. My patch should do a lot more to penalize that trick than Skip's. Both patches tokenize the de-obfuscated URL, so they're a wash in that respect. Skip's patch also exposes higher-level concepts to the classifier, like "non-standard port number". I don't see that often, but when I do it's usually in email from my work account (e.g., trying to get me to preview a pre-release site change, accessed via a non-standard port so it doesn't interfere with the production site). That's OK, though: *my* classifier will learn that's a ham clue in my email mix -- so it goes. Since everyone is getting some good out of Skip's changes (and I don't think his treatment of % escapes is making a difference), and also getting some good out of mine (which don't try to do anything except get some good of % escapes), combining the two will do better than either, or cancel each other out <0.5 wink>.
Tony> 3-way compare: Tony> filename: bases nntims pickskips Tony> ham:spam: 3668:4099 3668:4099 Tony> 3668:4099 Tony> fp total: 6 6 6 Tony> fp %: 0.16 0.16 0.16 Tony> fn total: 20 20 20 ... What do you use to generate these three-way comparisons? Skip
participants (3)
-
Skip Montanaro -
Tim Peters -
Tony Meyer