[Spambayes] Frequency distribution for wordinfo counts?
Skip Montanaro
skip at pobox.com
Mon Feb 23 16:05:24 EST 2004
Brad> I'd like to get feedback from folks on the distribution of nham
Brad> and nspam counts in their wordinfo databases.
...
Brad> Anyway, what I'm interested in is the number of tokens whose nspam
Brad> or nham count is greater than 255 vs the total number of tokens
Brad> and ham and spam count.
I'm coming into this late, but thought I'd post my numbers. As far as I
know, I'm the only person using train-to-exhaustion at the moment. That
probably skews my numbers, so maybe they'll be of interest.
When I run tte.py it reports 764 each of ham and spam (it always trains in
pairs and ignores any extras in the larger corpus). When I ask spamcounts
for the nham and nspam values from the database, it reports 301 and 237,
respectively. Some messages are never trained on since they score correctly
from the get-go. Other messages are trained on multiple times, currently
none more than four times. My current tte.db file is 2.6MB.
Appended are my counts/per cents.
Skip
------------------------------------------------------------------------------
ham dist
count freq %
0 22140 48.09907
1 18410 39.99566
2 2456 5.33565
3 935 2.03128
4 501 1.08842
5 295 0.64089
6 248 0.53878
7 162 0.35194
8 124 0.26939
9 89 0.19335
10 73 0.15859
11 65 0.14121
12 64 0.13904
13 38 0.08255
14 37 0.08038
15 23 0.04997
16 30 0.06517
17 24 0.05214
18 20 0.04345
19 20 0.04345
20 13 0.02824
21 12 0.02607
22 6 0.01303
23 12 0.02607
24 15 0.03259
25 14 0.03041
26 11 0.02390
27 7 0.01521
28 11 0.02390
29 10 0.02172
30 1 0.00217
31 9 0.01955
32 6 0.01303
33 2 0.00434
34 4 0.00869
35 4 0.00869
36 3 0.00652
37 2 0.00434
38 3 0.00652
39 5 0.01086
40 3 0.00652
41 3 0.00652
42 3 0.00652
43 4 0.00869
44 3 0.00652
45 8 0.01738
46 4 0.00869
47 4 0.00869
48 1 0.00217
49 1 0.00217
50 3 0.00652
51 2 0.00434
52 3 0.00652
53 2 0.00434
54 2 0.00434
55 2 0.00434
56 4 0.00869
57 5 0.01086
58 2 0.00434
59 1 0.00217
60 1 0.00217
61 1 0.00217
63 1 0.00217
64 5 0.01086
65 1 0.00217
66 1 0.00217
67 1 0.00217
69 1 0.00217
70 1 0.00217
72 2 0.00434
74 1 0.00217
75 1 0.00217
76 3 0.00652
77 1 0.00217
78 1 0.00217
79 1 0.00217
81 1 0.00217
90 1 0.00217
92 1 0.00217
94 2 0.00434
98 1 0.00217
103 2 0.00434
104 1 0.00217
105 1 0.00217
107 1 0.00217
109 1 0.00217
114 1 0.00217
122 1 0.00217
131 1 0.00217
132 1 0.00217
138 2 0.00434
139 1 0.00217
143 1 0.00217
152 2 0.00434
157 1 0.00217
168 1 0.00217
179 1 0.00217
189 1 0.00217
194 1 0.00217
212 2 0.00434
217 1 0.00217
220 1 0.00217
221 4 0.00869
224 1 0.00217
226 1 0.00217
233 1 0.00217
236 2 0.00434
237 3 0.00652
spam dist
count freq %
0 17009 36.95199
1 23725 51.54247
2 2316 5.03150
3 875 1.90093
4 484 1.05149
5 315 0.68434
6 231 0.50185
7 140 0.30415
8 107 0.23246
9 116 0.25201
10 69 0.14990
11 77 0.16728
12 45 0.09776
13 37 0.08038
14 33 0.07169
15 32 0.06952
16 22 0.04779
17 34 0.07386
18 16 0.03476
19 27 0.05866
20 12 0.02607
21 17 0.03693
22 17 0.03693
23 12 0.02607
24 11 0.02390
25 9 0.01955
26 11 0.02390
27 11 0.02390
28 9 0.01955
29 6 0.01303
30 10 0.02172
31 8 0.01738
32 8 0.01738
33 2 0.00434
34 2 0.00434
35 4 0.00869
36 8 0.01738
37 4 0.00869
38 9 0.01955
39 4 0.00869
40 1 0.00217
41 4 0.00869
42 3 0.00652
43 3 0.00652
44 6 0.01303
45 3 0.00652
46 2 0.00434
47 5 0.01086
48 7 0.01521
49 4 0.00869
50 8 0.01738
51 3 0.00652
52 2 0.00434
53 3 0.00652
54 3 0.00652
55 3 0.00652
57 4 0.00869
59 1 0.00217
60 2 0.00434
61 3 0.00652
62 1 0.00217
63 2 0.00434
64 1 0.00217
66 2 0.00434
67 1 0.00217
69 1 0.00217
70 3 0.00652
71 1 0.00217
72 1 0.00217
73 1 0.00217
74 1 0.00217
75 1 0.00217
77 2 0.00434
78 1 0.00217
81 2 0.00434
82 2 0.00434
83 1 0.00217
85 1 0.00217
90 1 0.00217
105 1 0.00217
109 2 0.00434
111 1 0.00217
112 2 0.00434
113 1 0.00217
115 2 0.00434
120 1 0.00217
121 1 0.00217
125 2 0.00434
128 1 0.00217
130 1 0.00217
132 1 0.00217
135 1 0.00217
155 1 0.00217
159 1 0.00217
168 2 0.00434
171 1 0.00217
173 1 0.00217
184 1 0.00217
185 2 0.00434
190 1 0.00217
197 1 0.00217
202 2 0.00434
213 1 0.00217
226 1 0.00217
234 1 0.00217
250 1 0.00217
259 1 0.00217
260 1 0.00217
263 1 0.00217
283 1 0.00217
284 1 0.00217
286 1 0.00217
287 4 0.00869
296 1 0.00217
299 2 0.00434
301 3 0.00652
More information about the Spambayes
mailing list