Mailman 3 October 2003 - spambayes-dev

Next Release
by Meyer, Tony Oct. 31, 2003

Oct. 31, 2003

Oh well, it's the thought that counts. I didn't manage to get the release after all; however, I did manage to get the What's New file done, and the release candidate works for me and for Neale. Anthony/Richie - over to you. If anyone feels like it, it occured to me (today) that there are probably a couple of fixes (the which_database ones, particularly) that could be copied over to the 1_0 branch, but it doesn't matter much if they aren't. Have fun :) =Tony Meyer

5 4

Re: [spambayes-dev] accented characters in subject
by Serge Orlov Oct. 31, 2003

Oct. 31, 2003

Hello, Paul, > Further to my email a few weeks ago, as well as punctuation in the middle of > words in the subject line I am also starting to get spam using umlauts, > accents and that kind of thing in the subject line. And they usually appear as non-converted 8 bit characters, right? If so, I suggest generating header:8bit token. This token will be a good spam clue since spamware quite often don't care about standards. > > I am not sure what the conclusion was re the efficacy of removing > punctuation but if it is an improvement then folding extended characters > into plain ASCII might also help. What is the difference between this trick and V1agra? Spambayes coped with V1agra quite well. I keep seeing it from time to time, but spambayes finds enough clues to overcome V1agra.

1 0

accented characters in subject
by Paul Sorenson Oct. 31, 2003

Oct. 31, 2003

Further to my email a few weeks ago, as well as punctuation in the middle of words in the subject line I am also starting to get spam using umlauts, accents and that kind of thing in the subject line. I am not sure what the conclusion was re the efficacy of removing punctuation but if it is an improvement then folding extended characters into plain ASCII might also help. cheers

2 1

auto-responder text for spambayes-bugs
by Mark Hammond Oct. 31, 2003

Oct. 31, 2003

I've seen a number of people mailing spambayes-bugs, trying to report a bug. I've added an autoresponder message to this list to try and direct people to SourceForge. The only poster to this list is generally noreply(a)sourceforge.net, and that address should only get one mail every 3 months - so I don't see any real problems - but if anyone else does, let me know (or better yet, fix it - password is the same as all other SB lists - let me know if you want to know what that is.) Mark.

1 0

RE: [spambayes-dev] removing punctuation redux
by Tony Meyer Oct. 29, 2003

Oct. 29, 2003

Hi Remi, > If you tell me the command to use I will do a run during my > dinner time. > > I'm a bit confused with split.py splitndir, tim timcv etc..... You should be able to copy Skip's script (at the bottom of his message) and use that (you're on a *nix system, yes?). Otherwise, what you basically do is (assumes you have the test directories all set up like in README-DEVEL): 1. Run "timcv.py -n 10 -s 12345 > std.txt" (10 is the number of directories you have) 2. Make the change you want to test (for Skip's change the punctation option). 3. Run "timcv.py -n 10 -s 12345 > other.txt" (10 is the number of directories you have) 4. Run "rates.py std.txt" 5. Run "rates.py other.txt" 6. Run "cmp.py stds.txt others.txt > results.txt" The results.txt file will have the results that you're after. [Note, however, that given the number of times Tim has had to chastise <wink> me for poor testing methodology, this should be taken with a grain of salt.] =Tony Meyer

2 1

RE: [spambayes-dev] removing punctuation redux
by Tony Meyer Oct. 29, 2003

Oct. 29, 2003

[...] > Would someone please try this with their training database? [...] Only time for a quick test and no analysis, so someone else can do that. Here are my results (following Skip's recipe). stds.txt -> puncts.txt -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams -> <stat> tested 231 hams & 439 spams against 2079 hams & 3951 spams false positive percentages 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.433 0.433 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied 0.000 0.000 tied won 0 times tied 10 times lost 0 times total unique fp went from 1 to 1 tied mean fp % went from 0.04329004329 to 0.04329004329 tied false negative percentages 2.506 2.506 tied 1.822 1.822 tied 1.822 1.822 tied 1.367 1.367 tied 2.733 2.506 won -8.31% 1.595 1.367 won -14.29% 1.822 1.822 tied 2.506 2.506 tied 1.822 1.822 tied 1.822 1.822 tied won 2 times tied 8 times lost 0 times total unique fn went from 87 to 85 won -2.30% mean fn % went from 1.98177676537 to 1.93621867881 won -2.30% ham mean ham sdev 0.10 0.10 +0.00% 1.03 1.04 +0.97% 0.04 0.03 -25.00% 0.24 0.22 -8.33% 0.29 0.29 +0.00% 3.04 3.04 +0.00% 0.20 0.20 +0.00% 2.64 2.68 +1.52% 0.35 0.35 +0.00% 3.88 3.88 +0.00% 0.71 0.70 -1.41% 6.48 6.47 -0.15% 0.79 0.79 +0.00% 6.91 6.92 +0.14% 0.27 0.27 +0.00% 3.30 3.30 +0.00% 0.32 0.30 -6.25% 3.28 3.03 -7.62% 0.42 0.41 -2.38% 5.28 5.35 +1.33% ham mean and sdev for all runs 0.35 0.34 -2.86% 4.15 4.14 -0.24% spam mean spam sdev 95.71 95.70 -0.01% 17.16 17.17 +0.06% 96.97 96.99 +0.02% 14.91 14.78 -0.87% 97.15 97.15 +0.00% 13.95 13.98 +0.22% 96.64 96.69 +0.05% 14.09 14.05 -0.28% 96.06 96.09 +0.03% 17.51 17.44 -0.40% 96.97 97.04 +0.07% 13.81 13.71 -0.72% 96.58 96.60 +0.02% 15.17 15.13 -0.26% 96.52 96.52 +0.00% 15.96 15.93 -0.19% 96.28 96.27 -0.01% 15.34 15.36 +0.13% 96.95 96.94 -0.01% 14.76 14.74 -0.14% spam mean and sdev for all runs 96.58 96.60 +0.02% 15.32 15.28 -0.26% ham/spam mean difference: 96.23 96.26 +0.03 =Tony Meyer

1 0

removing punctuation redux
by Skip Montanaro Oct. 29, 2003

Oct. 29, 2003

Just as I was nodding off to sleep last night I realized I hadn't done the punctuation removal right. I was stripping punctuation after testing word length, so (for example) a nine-letter word containing four punctuation characters got evaluated as a "long" word. I redid it this morning so the punctuation stripping occurs before the length test and got different results (a bit better overall I think). Here's the summary (minus the first block). All rounds scored 500 against 4500 (using the same training sets as yesterday). stds.txt -> puncts.txt false positive percentages 0.200 0.200 tied 0.200 0.400 lost +100.00% 0.200 0.200 tied 0.400 0.400 tied 1.000 1.000 tied 0.400 0.400 tied 0.000 0.000 tied 0.200 0.200 tied 0.400 0.400 tied 0.400 0.400 tied won 0 times tied 9 times lost 1 times total unique fp went from 17 to 18 lost +5.88% mean fp % went from 0.34 to 0.36 lost +5.88% false negative percentages 2.000 2.000 tied 0.800 1.000 lost +25.00% 2.200 2.200 tied 2.200 2.400 lost +9.09% 2.200 2.000 won -9.09% 1.200 1.200 tied 2.000 1.800 won -10.00% 2.000 1.800 won -10.00% 1.600 1.600 tied 1.200 0.800 won -33.33% won 4 times tied 4 times lost 2 times total unique fn went from 87 to 84 won -3.45% mean fn % went from 1.74 to 1.68 won -3.45% ham mean ham sdev 4.70 4.35 -7.45% 14.04 13.40 -4.56% 6.12 6.06 -0.98% 16.43 16.55 +0.73% 4.14 4.01 -3.14% 12.86 12.62 -1.87% 4.64 4.56 -1.72% 14.35 14.23 -0.84% 5.49 5.31 -3.28% 17.21 16.99 -1.28% 5.21 5.03 -3.45% 16.10 15.81 -1.80% 3.74 3.62 -3.21% 12.19 12.03 -1.31% 3.38 3.30 -2.37% 11.70 11.51 -1.62% 5.50 5.46 -0.73% 17.07 17.09 +0.12% 5.44 5.27 -3.13% 15.85 15.57 -1.77% ham mean and sdev for all runs 4.84 4.70 -2.89% 14.93 14.74 -1.27% spam mean spam sdev 89.47 89.61 +0.16% 20.40 20.36 -0.20% 91.15 91.22 +0.08% 17.17 17.12 -0.29% 91.04 90.97 -0.08% 20.00 20.22 +1.10% 89.39 89.74 +0.39% 20.88 20.73 -0.72% 89.56 89.70 +0.16% 19.92 19.72 -1.00% 90.72 90.78 +0.07% 18.41 18.48 +0.38% 90.97 91.09 +0.13% 18.66 18.54 -0.64% 91.66 91.85 +0.21% 18.20 17.90 -1.65% 91.58 91.44 -0.15% 18.03 18.24 +1.16% 92.24 92.25 +0.01% 16.18 16.22 +0.25% spam mean and sdev for all runs 90.78 90.87 +0.10% 18.86 18.83 -0.16% ham/spam mean difference: 85.94 86.17 +0.23 The new context diff is attached. Would someone please try this with their training database? The shell script I used to run the test is #!/bin/sh # point SBDIR at Spambayes root directory SBDIR=$HOME/src/spambayes # everything below here should be okay TIMCV=$SBDIR/testtools/timcv.py RATE=$SBDIR/testtools/rates.py CMP=$SBDIR/testtools/cmp.py base=$1 trial=$2 BAYESCUSTOMIZE=$base.ini ; export BAYESCUSTOMIZE python $TIMCV -n 10 -s 12345 > $base.txt BAYESCUSTOMIZE=$trial.ini ; export BAYESCUSTOMIZE python $TIMCV -n 10 -s 12345 > $trial.txt python $RATE $base.txt python $RATE $trial.txt python $CMP ${base}s.txt ${trial}s.txt > $base-$trial.txt So I created an empty std.ini and a punct.ini with this: [Tokenizer] remove_punctuation: True and executed sh runtest std punct Thx, Skip

1 0

removing punctuation - no apparent benefit
by Skip Montanaro Oct. 28, 2003

Oct. 28, 2003

I tried removing punctuation from words as we discussed a week or so ago. No overall change, though per/hap.s we just ne.ed to see more mes*sages with em%bedded p-unctua-tion. In general, the ham means decreased slightly and the spam means increased slightly while the standard deviations of both groups decreased a little. stds.txt -> puncts.txt -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams -> <stat> tested 500 hams & 500 spams against 4500 hams & 4500 spams false positive percentages 0.200 0.200 tied 0.200 0.200 tied 0.200 0.200 tied 0.400 0.400 tied 1.000 1.000 tied 0.400 0.400 tied 0.000 0.000 tied 0.200 0.200 tied 0.400 0.400 tied 0.400 0.400 tied won 0 times tied 10 times lost 0 times total unique fp went from 17 to 17 tied mean fp % went from 0.34 to 0.34 tied false negative percentages 2.000 2.000 tied 0.800 0.800 tied 2.200 2.200 tied 2.200 2.200 tied 2.200 2.200 tied 1.200 1.200 tied 2.000 2.000 tied 2.000 2.000 tied 1.600 1.600 tied 1.200 1.200 tied won 0 times tied 10 times lost 0 times total unique fn went from 87 to 87 tied mean fn % went from 1.74 to 1.74 tied ham mean ham sdev 4.70 4.53 -3.62% 14.04 13.69 -2.49% 6.12 6.13 +0.16% 16.43 16.48 +0.30% 4.14 4.07 -1.69% 12.86 12.73 -1.01% 4.64 4.62 -0.43% 14.35 14.34 -0.07% 5.49 5.41 -1.46% 17.21 17.16 -0.29% 5.21 5.16 -0.96% 16.10 16.01 -0.56% 3.74 3.70 -1.07% 12.19 12.12 -0.57% 3.38 3.34 -1.18% 11.70 11.68 -0.17% 5.50 5.48 -0.36% 17.07 17.05 -0.12% 5.44 5.38 -1.10% 15.85 15.82 -0.19% ham mean and sdev for all runs 4.84 4.78 -1.24% 14.93 14.86 -0.47% spam mean spam sdev 89.47 89.57 +0.11% 20.40 20.37 -0.15% 91.15 91.24 +0.10% 17.17 17.02 -0.87% 91.04 91.03 -0.01% 20.00 20.03 +0.15% 89.39 89.54 +0.17% 20.88 20.76 -0.57% 89.56 89.63 +0.08% 19.92 19.91 -0.05% 90.72 90.77 +0.06% 18.41 18.36 -0.27% 90.97 91.03 +0.07% 18.66 18.60 -0.32% 91.66 91.78 +0.13% 18.20 18.08 -0.66% 91.58 91.61 +0.03% 18.03 18.03 +0.00% 92.24 92.30 +0.07% 16.18 16.18 +0.00% spam mean and sdev for all runs 90.78 90.85 +0.08% 18.86 18.81 -0.27% ham/spam mean difference: 85.94 86.07 +0.13 The training database I used is just my current collection of ham and spam I use every day. Attached is the context diff to tokenizer.py and Options.py should anyone care to check my work for mistakes or tweak it. Skip

1 0

Empty Spam Folder menu item
by Kenny Pitt Oct. 28, 2003

Oct. 28, 2003

Feature Request #788755 asks for a menu item to empty the spam folder, and I have seen this requested several times on the lists. I've come up with an initial patch to implement this as a menu item on the SpamBayes dropdown. It works very much like the Empty "Deleted Items" Folder context menu item, which means it prompts with a Yes/No message box before deleting, and just does nothing if the spam folder is already empty. I was hoping some of you here would be willing to review and/or test this before I submit it as a patch on SourceForge. BTW, what is the recommended way to submit a patch for a feature request? There is no obvious way to attach a file when adding a comment. -- Kenny Pitt

2 2

Auto-generation of options documentation
by Neale Pickett Oct. 28, 2003

Oct. 28, 2003

I whipped a little something together to generate a document about the possible options for the .ini file. Here's the output: http://woozle.org/~neale/f.html Here's the source: import cgi from spambayes import Options for k,v in Options.defaults.iteritems(): print '<h1>[%s]</h1>' % k print '<dl>' for item in v: print '<dt>%s</dt>' % item[0] print '<dd>' print '<p><b>%s</b></p>' % item[1] print '<p>%s</p>' % item[3] print '<p>Default: %s</p>' % cgi.escape(str(item[2])) print '</dd>' print '</dl>' print '<hr>' Would a souped-up version of this be a useful addition to the website? I mean, we've already gone to the trouble to document the entire options file... Neale (Not the best at web page design)

2 1