[spambayes-dev] spammy subject lines

Tony Meyer tameyer at ihug.co.nz
Mon Oct 13 20:47:07 EDT 2003


> Never use timtest.  It's slow and too hard to interpret 
> (we've been thru this before, right?).

Well, you've never been that definite about it.  More of a *recommendation*
to use timcv, which was the reason that I did at first :)

> I don't know what mkgraph.py does; looks like it produces 
> input for some graph-drawing package.  Unless the code has 
> rotted due to disuse, the test drivers automatically produce 
> ASCII-art histograms, controlled by the nbuckets and 
> show_histograms options.  IIRC, you cut-and-paste them out of 
> the full output.

<click>.  Ah yes, I remember those now :)

> cmp.py produces (unless the code has rotted due to disuse) an 
> account of how many runs lost, won and tied.  table.py is 
> much more telegraphic, and more useful for getting a quick 
> feel for the relative results across several alternatives.  
> cmp.py is only concerned with producing "before" and "after" 
> statistics for a single change, and gives more detail about 
> that change than table.py produces.

So, I (and anyone else ;) should do timcv.py, rates.py and cmp.py and post
the results of that?  Along with any necessary histograms from the timcv.py
output, then.  So here's a new attempt (the first data was, as you said in
your next message, basically spam and 'hard' ham, this data is spam and hard
and 'soft' ham).

cv_octs.txt -> cv_oct_subjs.txt
-> <stat> tested 488 hams & 897 spams against 1824 hams & 3501 spams
-> <stat> tested 462 hams & 863 spams against 1850 hams & 3535 spams
-> <stat> tested 475 hams & 863 spams against 1837 hams & 3535 spams
-> <stat> tested 430 hams & 887 spams against 1882 hams & 3511 spams
-> <stat> tested 457 hams & 888 spams against 1855 hams & 3510 spams
-> <stat> tested 488 hams & 897 spams against 1824 hams & 3501 spams
-> <stat> tested 462 hams & 863 spams against 1850 hams & 3535 spams
-> <stat> tested 475 hams & 863 spams against 1837 hams & 3535 spams
-> <stat> tested 430 hams & 887 spams against 1882 hams & 3511 spams
-> <stat> tested 457 hams & 888 spams against 1855 hams & 3510 spams

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.219  0.219  tied

won   0 times
tied  5 times
lost  0 times

total unique fp went from 1 to 1 tied
mean fp % went from 0.0437636761488 to 0.0437636761488 tied

false negative percentages
    2.007  2.007  tied
    1.390  1.390  tied
    1.622  1.622  tied
    2.029  1.917  won     -5.52%
    2.703  2.477  won     -8.36%

won   2 times
tied  3 times
lost  0 times

total unique fn went from 86 to 83 won     -3.49%
mean fn % went from 1.95029003772 to 1.88269707836 won     -3.47%

ham mean                     ham sdev
   0.57    0.58   +1.75%        4.63    4.77   +3.02%
   0.08    0.07  -12.50%        1.20    1.01  -15.83%
   0.36    0.29  -19.44%        3.61    3.23  -10.53%
   0.08    0.11  +37.50%        0.89    1.18  +32.58%
   0.72    0.76   +5.56%        6.80    7.06   +3.82%

ham mean and sdev for all runs
   0.37    0.37   +0.00%        4.10    4.16   +1.46%

spam mean                    spam sdev
  96.43   96.44   +0.01%       15.89   15.89   +0.00%
  97.01   97.07   +0.06%       13.79   13.70   -0.65%
  97.14   97.16   +0.02%       14.05   14.02   -0.21%
  96.52   96.56   +0.04%       15.65   15.52   -0.83%
  95.53   95.63   +0.10%       17.47   17.31   -0.92%

spam mean and sdev for all runs
  96.52   96.57   +0.05%       15.46   15.37   -0.58%

ham/spam mean difference: 96.15 96.20 +0.05

I *think* ;) that this is back to a slight win for the change...

=Tony Meyer




More information about the spambayes-dev mailing list