[spambayes-dev] Mine_received_headers

Tony Meyer ta-meyer at ihug.co.nz
Wed Jan 7 19:10:25 EST 2004


While I was testing, I figured I should do this one, too.  This gives me
terrible results!  I haven't tested with the old ('reverse') method, so I
don't know if that would have made a difference or not.

Without bigrams:

-> <stat> tested 357 hams & 395 spams against 3311 hams & 3704 spams
[all other stat lines in the message snipped - they're all basically this]

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.246  0.246  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.557  0.557  tied
    0.559  0.279  won    -50.09%
    0.287  0.287  tied
    0.000  0.000  tied

won   1 times
tied  9 times
lost  0 times

total unique fp went from 6 to 5 won    -16.67%
mean fp % went from 0.164881884948 to 0.136948924055 won    -16.94%

false negative percentages
    0.253  0.506  lost  +100.00%
    0.781  1.042  lost   +33.42%
    0.462  1.155  lost  +150.00%
    0.756  0.756  tied
    0.243  0.243  tied
    0.247  0.494  lost  +100.00%
    0.240  1.202  lost  +400.83%
    0.494  1.235  lost  +150.00%
    0.973  1.217  lost   +25.08%
    0.454  0.680  lost   +49.78%

won   0 times
tied  2 times
lost  8 times

total unique fn went from 20 to 35 lost   +75.00%
mean fn % went from 0.490257037938 to 0.852825140424 lost   +73.95%

ham mean                     ham sdev
   1.18    0.93  -21.19%        7.76    7.22   -6.96%
   0.99    0.91   -8.08%        6.64    6.88   +3.61%
   0.84    0.61  -27.38%        6.14    5.32  -13.36%
   1.99    2.05   +3.02%        9.46    9.97   +5.39%
   0.49    0.44  -10.20%        3.59    3.43   -4.46%
   0.85    0.89   +4.71%        5.45    6.08  +11.56%
   1.16    1.28  +10.34%        9.30    9.84   +5.81%
   1.20    0.99  -17.50%        8.13    7.13  -12.30%
   1.55    1.72  +10.97%        8.05    9.18  +14.04%
   0.47    0.43   -8.51%        3.22    3.53   +9.63%

ham mean and sdev for all runs
   1.08    1.03   -4.63%        7.13    7.26   +1.82%

spam mean                    spam sdev
  98.75   98.79   +0.04%        8.72    8.79   +0.80%
  97.67   96.87   -0.82%       11.26   13.66  +21.31%
  98.08   96.97   -1.13%       10.12   13.46  +33.00%
  98.16   97.75   -0.42%       10.19   11.56  +13.44%
  98.35   98.28   -0.07%        8.77    9.05   +3.19%
  98.45   98.10   -0.36%        8.97   10.13  +12.93%
  98.35   97.70   -0.66%        9.73   12.49  +28.37%
  98.25   97.91   -0.35%        9.16   11.41  +24.56%
  97.93   97.17   -0.78%       11.99   13.97  +16.51%
  98.92   98.66   -0.26%        7.62    9.34  +22.57%

spam mean and sdev for all runs
  98.30   97.82   -0.49%        9.72   11.55  +18.83%

ham/spam mean difference: 97.22 96.79 -0.43

With bigrams:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.246  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied
    0.279  0.279  tied
    0.000  0.000  tied
    0.000  0.287  lost  +(was 0)
    0.000  0.000  tied

won   0 times
tied  8 times
lost  2 times

total unique fp went from 1 to 3 lost  +200.00%
mean fp % went from 0.0278551532033 to 0.0811608099572 lost  +191.37%

false negative percentages
    0.253  0.506  lost  +100.00%
    1.042  1.302  lost   +24.95%
    0.693  1.155  lost   +66.67%
    0.252  0.504  lost  +100.00%
    0.728  0.243  won    -66.62%
    0.000  0.247  lost  +(was 0)
    0.481  0.962  lost  +100.00%
    0.494  1.235  lost  +150.00%
    0.730  1.217  lost   +66.71%
    0.227  0.227  tied

won   1 times
tied  1 times
lost  8 times

total unique fn went from 20 to 31 lost   +55.00%
mean fn % went from 0.489899714703 to 0.759596596728 lost   +55.05%

ham mean                     ham sdev
   0.95    0.81  -14.74%        6.64    6.30   -5.12%
   0.83    0.75   -9.64%        5.53    5.65   +2.17%
   0.49    0.36  -26.53%        4.08    3.74   -8.33%
   1.53    1.62   +5.88%        8.16    8.47   +3.80%
   0.30    0.29   -3.33%        3.25    3.01   -7.38%
   0.70    0.64   -8.57%        5.27    5.09   -3.42%
   0.85    0.84   -1.18%        7.11    7.25   +1.97%
   0.93    0.68  -26.88%        7.23    5.83  -19.36%
   0.90    1.04  +15.56%        6.47    7.26  +12.21%
   0.41    0.38   -7.32%        4.07    4.00   -1.72%

ham mean and sdev for all runs
   0.80    0.75   -6.25%        6.01    5.93   -1.33%

spam mean                    spam sdev
  98.71   98.67   -0.04%        7.83    8.75  +11.75%
  97.38   96.86   -0.53%       12.55   14.27  +13.71%
  97.78   96.99   -0.81%       11.09   13.46  +21.37%
  97.89   97.61   -0.29%       10.49   11.33   +8.01%
  97.90   97.78   -0.12%       10.03   10.13   +1.00%
  98.32   97.83   -0.50%        8.63   10.20  +18.19%
  98.19   97.89   -0.31%       10.21   11.44  +12.05%
  97.68   97.61   -0.07%       10.99   11.94   +8.64%
  97.86   97.13   -0.75%       11.56   13.79  +19.29%
  98.73   98.71   -0.02%        7.57    8.14   +7.53%

spam mean and sdev for all runs
  98.05   97.71   -0.35%       10.20   11.51  +12.84%

ham/spam mean difference: 97.25 96.96 -0.29

Table:

filename:        bases   mine_recs     basebis mine_rec_bis
ham:spam:    3668:4099   3668:4099   3668:4099   3668:4099
fp total:            6           5           1           3
fp %:             0.16        0.14        0.03        0.08
fn total:           20          35          20          31
fn %:             0.49        0.85        0.49        0.76
unsure t:          178         198         207         205
unsure %:         2.29        2.55        2.67        2.64
real cost:     $115.60     $124.60      $71.40     $102.00
best cost:      $93.00     $119.80      $65.60      $78.20
h mean:           1.08        1.03        0.80        0.75
h sdev:           7.13        7.26        6.01        5.93
s mean:          98.30       97.82       98.05       97.71
s sdev:           9.72       11.55       10.20       11.51
mean diff:       97.22       96.79       97.25       96.96
k:                5.77        5.15        6.00        5.56

=Tony Meyer




More information about the spambayes-dev mailing list