[Spambayes] Frequency distribution for wordinfo counts?

Skip Montanaro skip at pobox.com
Mon Feb 23 16:05:24 EST 2004


    Brad> I'd like to get feedback from folks on the distribution of nham
    Brad> and nspam counts in their wordinfo databases.

    ...

    Brad> Anyway, what I'm interested in is the number of tokens whose nspam
    Brad> or nham count is greater than 255 vs the total number of tokens
    Brad> and ham and spam count.

I'm coming into this late, but thought I'd post my numbers.  As far as I
know, I'm the only person using train-to-exhaustion at the moment.  That
probably skews my numbers, so maybe they'll be of interest.

When I run tte.py it reports 764 each of ham and spam (it always trains in
pairs and ignores any extras in the larger corpus).  When I ask spamcounts
for the nham and nspam values from the database, it reports 301 and 237,
respectively.  Some messages are never trained on since they score correctly
from the get-go.  Other messages are trained on multiple times, currently
none more than four times.  My current tte.db file is 2.6MB.

Appended are my counts/per cents.

Skip

------------------------------------------------------------------------------

ham dist

count     freq  %
   0     22140  48.09907
   1     18410  39.99566
   2      2456   5.33565
   3       935   2.03128
   4       501   1.08842
   5       295   0.64089
   6       248   0.53878
   7       162   0.35194
   8       124   0.26939
   9        89   0.19335
  10        73   0.15859
  11        65   0.14121
  12        64   0.13904
  13        38   0.08255
  14        37   0.08038
  15        23   0.04997
  16        30   0.06517
  17        24   0.05214
  18        20   0.04345
  19        20   0.04345
  20        13   0.02824
  21        12   0.02607
  22         6   0.01303
  23        12   0.02607
  24        15   0.03259
  25        14   0.03041
  26        11   0.02390
  27         7   0.01521
  28        11   0.02390
  29        10   0.02172
  30         1   0.00217
  31         9   0.01955
  32         6   0.01303
  33         2   0.00434
  34         4   0.00869
  35         4   0.00869
  36         3   0.00652
  37         2   0.00434
  38         3   0.00652
  39         5   0.01086
  40         3   0.00652
  41         3   0.00652
  42         3   0.00652
  43         4   0.00869
  44         3   0.00652
  45         8   0.01738
  46         4   0.00869
  47         4   0.00869
  48         1   0.00217
  49         1   0.00217
  50         3   0.00652
  51         2   0.00434
  52         3   0.00652
  53         2   0.00434
  54         2   0.00434
  55         2   0.00434
  56         4   0.00869
  57         5   0.01086
  58         2   0.00434
  59         1   0.00217
  60         1   0.00217
  61         1   0.00217
  63         1   0.00217
  64         5   0.01086
  65         1   0.00217
  66         1   0.00217
  67         1   0.00217
  69         1   0.00217
  70         1   0.00217
  72         2   0.00434
  74         1   0.00217
  75         1   0.00217
  76         3   0.00652
  77         1   0.00217
  78         1   0.00217
  79         1   0.00217
  81         1   0.00217
  90         1   0.00217
  92         1   0.00217
  94         2   0.00434
  98         1   0.00217
 103         2   0.00434
 104         1   0.00217
 105         1   0.00217
 107         1   0.00217
 109         1   0.00217
 114         1   0.00217
 122         1   0.00217
 131         1   0.00217
 132         1   0.00217
 138         2   0.00434
 139         1   0.00217
 143         1   0.00217
 152         2   0.00434
 157         1   0.00217
 168         1   0.00217
 179         1   0.00217
 189         1   0.00217
 194         1   0.00217
 212         2   0.00434
 217         1   0.00217
 220         1   0.00217
 221         4   0.00869
 224         1   0.00217
 226         1   0.00217
 233         1   0.00217
 236         2   0.00434
 237         3   0.00652

spam dist

count     freq  %
   0     17009  36.95199
   1     23725  51.54247
   2      2316   5.03150
   3       875   1.90093
   4       484   1.05149
   5       315   0.68434
   6       231   0.50185
   7       140   0.30415
   8       107   0.23246
   9       116   0.25201
  10        69   0.14990
  11        77   0.16728
  12        45   0.09776
  13        37   0.08038
  14        33   0.07169
  15        32   0.06952
  16        22   0.04779
  17        34   0.07386
  18        16   0.03476
  19        27   0.05866
  20        12   0.02607
  21        17   0.03693
  22        17   0.03693
  23        12   0.02607
  24        11   0.02390
  25         9   0.01955
  26        11   0.02390
  27        11   0.02390
  28         9   0.01955
  29         6   0.01303
  30        10   0.02172
  31         8   0.01738
  32         8   0.01738
  33         2   0.00434
  34         2   0.00434
  35         4   0.00869
  36         8   0.01738
  37         4   0.00869
  38         9   0.01955
  39         4   0.00869
  40         1   0.00217
  41         4   0.00869
  42         3   0.00652
  43         3   0.00652
  44         6   0.01303
  45         3   0.00652
  46         2   0.00434
  47         5   0.01086
  48         7   0.01521
  49         4   0.00869
  50         8   0.01738
  51         3   0.00652
  52         2   0.00434
  53         3   0.00652
  54         3   0.00652
  55         3   0.00652
  57         4   0.00869
  59         1   0.00217
  60         2   0.00434
  61         3   0.00652
  62         1   0.00217
  63         2   0.00434
  64         1   0.00217
  66         2   0.00434
  67         1   0.00217
  69         1   0.00217
  70         3   0.00652
  71         1   0.00217
  72         1   0.00217
  73         1   0.00217
  74         1   0.00217
  75         1   0.00217
  77         2   0.00434
  78         1   0.00217
  81         2   0.00434
  82         2   0.00434
  83         1   0.00217
  85         1   0.00217
  90         1   0.00217
 105         1   0.00217
 109         2   0.00434
 111         1   0.00217
 112         2   0.00434
 113         1   0.00217
 115         2   0.00434
 120         1   0.00217
 121         1   0.00217
 125         2   0.00434
 128         1   0.00217
 130         1   0.00217
 132         1   0.00217
 135         1   0.00217
 155         1   0.00217
 159         1   0.00217
 168         2   0.00434
 171         1   0.00217
 173         1   0.00217
 184         1   0.00217
 185         2   0.00434
 190         1   0.00217
 197         1   0.00217
 202         2   0.00434
 213         1   0.00217
 226         1   0.00217
 234         1   0.00217
 250         1   0.00217
 259         1   0.00217
 260         1   0.00217
 263         1   0.00217
 283         1   0.00217
 284         1   0.00217
 286         1   0.00217
 287         4   0.00869
 296         1   0.00217
 299         2   0.00434
 301         3   0.00652




More information about the Spambayes mailing list