[Spambayes] timcombine results (long)
Brad Clements
bkc@murkworks.com
Thu, 10 Oct 2002 10:08:41 -0400
I re-ran tests on my corpus using this morning's checkout.
The only difference between the two runs was
use_tim_combining: (false first, then true)
Note, I left spam_cutoff at 0.50 for both runs!
[bkc@strader2 spambayes]$ more bayescustomize.ini
[Tokenizer]
mine_received_headers: True
[Classifier]
use_central_limit = False
use_central_limit2 = False
use_central_limit3 = False
zscore_ratio_cutoff: 1.9
use_tim_combining: False
[TestDriver]
spam_cutoff: 0.50
show_false_negatives: True
nbuckets: 100
show_spam_lo: 0.0
show_spam_hi: 0.45
save_trained_pickles: True
save_histogram_pickles: True
Histogram from from use_tim_combining: false
-> <stat> Ham scores for all runs: 13000 items; mean 25.35; sdev 6.95
-> <stat> min 0.771618; median 24.6878; max 78.0095
* = 20 items
0 12 *
1 12 *
2 1 *
3 7 *
4 12 *
5 12 *
6 29 **
7 13 *
8 32 **
9 32 **
10 45 ***
11 35 **
12 80 ****
13 122 *******
14 130 *******
15 191 **********
16 230 ************
17 300 ***************
18 369 *******************
19 533 ***************************
20 701 ************************************
21 839 ******************************************
22 946 ************************************************
23 998 **************************************************
24 1165 ***********************************************************
25 975 *************************************************
26 859 *******************************************
27 780 ***************************************
28 659 *********************************
29 541 ****************************
30 388 ********************
31 299 ***************
32 267 **************
33 197 **********
34 168 *********
35 144 ********
36 135 *******
37 102 ******
38 96 *****
39 84 *****
40 51 ***
41 64 ****
42 44 ***
43 42 ***
44 39 **
45 34 **
46 17 *
47 28 **
48 24 **
49 19 *
50 26 **
51 7 *
52 15 *
53 12 *
54 6 *
55 5 *
56 11 *
57 3 *
58 3 *
59 2 *
60 0
61 2 *
62 1 *
63 0
64 2 *
65 1 *
66 0
67 1 *
68 0
69 0
70 0
71 0
72 0
73 0
74 0
75 0
76 0
77 0
78 1 *
79 0
80 0
81 0
82 0
83 0
84 0
85 0
86 0
87 0
88 0
89 0
90 0
91 0
92 0
93 0
94 0
95 0
96 0
97 0
98 0
99 0
-> <stat> Spam scores for all runs: 13000 items; mean 81.18; sdev 7.55
-> <stat> min 34.1005; median 82.5437; max 99.5356
* = 17 items
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 0
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 0
20 0
21 0
22 0
23 0
24 0
25 0
26 0
27 0
28 0
29 0
30 0
31 0
32 0
33 0
34 1 *
35 0
36 1 *
37 0
38 0
39 1 *
40 0
41 0
42 0
43 3 *
44 1 *
45 5 *
46 2 *
47 1 *
48 5 *
49 0
50 7 *
51 9 *
52 7 *
53 12 *
54 18 **
55 18 **
56 23 **
57 22 **
58 30 **
59 38 ***
60 37 ***
61 49 ***
62 74 *****
63 56 ****
64 102 ******
65 83 *****
66 100 ******
67 102 ******
68 124 ********
69 160 **********
70 191 ************
71 228 **************
72 211 *************
73 261 ****************
74 318 *******************
75 312 *******************
76 411 *************************
77 413 *************************
78 497 ******************************
79 627 *************************************
80 745 ********************************************
81 780 **********************************************
82 861 ***************************************************
83 991 ***********************************************************
84 903 ******************************************************
85 860 ***************************************************
86 771 **********************************************
87 622 *************************************
88 506 ******************************
89 510 ******************************
90 230 **************
91 158 **********
92 142 *********
93 112 *******
94 79 *****
95 52 ****
96 38 ***
97 14 *
98 18 **
99 48 ***
-> best cutoff for all runs: 0.53
-> with weighted total 1*50 fp + 43 fn = 93
-> fp rate 0.385% fn rate 0.331%
-> matched at 0.54 with 38 fp & 55 fn; fp rate 0.292%; fn rate 0.423%
saving ham histogram pickle to class_hamhist.pik
saving spam histogram pickle to class_spamhist.pik
And now, histogram from timcombine true
-> <stat> Ham scores for all runs: 13000 items; mean 11.93; sdev 8.33
-> <stat> min 0.584578; median 9.92718; max 87.3273
* = 18 items
0 61 ****
1 220 *************
2 233 *************
3 398 ***********************
4 670 **************************************
5 811 **********************************************
6 930 ****************************************************
7 1080 ************************************************************
8 1081 *************************************************************
9 1088 *************************************************************
10 871 *************************************************
11 788 ********************************************
12 702 ***************************************
13 629 ***********************************
14 558 *******************************
15 419 ************************
16 337 *******************
17 268 ***************
18 229 *************
19 195 ***********
20 146 *********
21 141 ********
22 125 *******
23 105 ******
24 102 ******
25 75 *****
26 50 ***
27 60 ****
28 47 ***
29 58 ****
30 48 ***
31 49 ***
32 31 **
33 35 **
34 19 **
35 30 **
36 34 **
37 21 **
38 9 *
39 19 **
40 13 *
41 25 **
42 10 *
43 10 *
44 12 *
45 14 *
46 12 *
47 11 *
48 12 *
49 11 *
50 13 *
51 13 *
52 4 *
53 7 *
54 8 *
55 5 *
56 8 *
57 3 *
58 4 *
59 3 *
60 5 *
61 4 *
62 3 *
63 1 *
64 3 *
65 3 *
66 0
67 2 *
68 2 *
69 0
70 0
71 0
72 0
73 2 *
74 1 *
75 2 *
76 0
77 0
78 0
79 0
80 0
81 0
82 0
83 1 *
84 0
85 0
86 0
87 1 *
88 0
89 0
90 0
91 0
92 0
93 0
94 0
95 0
96 0
97 0
98 0
99 0
-> <stat> Spam scores for all runs: 13000 items; mean 90.62; sdev 7.36
-> <stat> min 11.1229; median 92.6441; max 99.5389
* = 21 items
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 1 *
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 1 *
20 0
21 1 *
22 0
23 0
24 0
25 0
26 0
27 0
28 0
29 0
30 0
31 0
32 1 *
33 0
34 0
35 1 *
36 0
37 1 *
38 0
39 1 *
40 0
41 2 *
42 5 *
43 1 *
44 1 *
45 0
46 3 *
47 1 *
48 0
49 0
50 2 *
51 7 *
52 3 *
53 3 *
54 6 *
55 3 *
56 7 *
57 13 *
58 6 *
59 11 *
60 13 *
61 16 *
62 11 *
63 18 *
64 22 **
65 20 *
66 28 **
67 33 **
68 24 **
69 36 **
70 55 ***
71 39 **
72 55 ***
73 77 ****
74 59 ***
75 69 ****
76 93 *****
77 100 *****
78 110 ******
79 152 ********
80 156 ********
81 172 *********
82 210 **********
83 193 **********
84 242 ************
85 278 **************
86 313 ***************
87 393 *******************
88 477 ***********************
89 608 *****************************
90 689 *********************************
91 950 **********************************************
92 1131 ******************************************************
93 1278 *************************************************************
94 1244 ************************************************************
95 945 *********************************************
96 902 *******************************************
97 1056 ***************************************************
98 604 *****************************
99 48 ***
-> best cutoff for all runs: 0.57
-> with weighted total 1*40 fp + 51 fn = 91
-> fp rate 0.308% fn rate 0.392%
saving ham histogram pickle to class_hamhist.pik
saving spam histogram pickle to class_spamhist.pik
And rates cmp.py
results/timcombinefalses.txt -> results/timcombinetrues.txt
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
<snip more lines like above>
false positive percentages
1.077 1.077 tied
0.769 0.769 tied
0.769 0.769 tied
0.923 0.923 tied
0.769 0.769 tied
0.538 0.538 tied
0.538 0.538 tied
0.692 0.692 tied
0.769 0.769 tied
0.692 0.692 tied
won 0 times
tied 10 times
lost 0 times
total unique fp went from 98 to 98 tied
mean fp % went from 0.753846153846 to 0.753846153846 tied
false negative percentages
0.154 0.154 tied
0.154 0.154 tied
0.231 0.231 tied
0.077 0.077 tied
0.000 0.000 tied
0.231 0.231 tied
0.231 0.231 tied
0.077 0.077 tied
0.154 0.154 tied
0.231 0.231 tied
won 0 times
tied 10 times
lost 0 times
total unique fn went from 20 to 20 tied
mean fn % went from 0.153846153846 to 0.153846153846 tied
ham mean ham sdev
25.47 12.23 -51.98% 7.31 9.02 +23.39%
25.37 12.04 -52.54% 7.07 8.57 +21.22%
25.56 12.08 -52.74% 6.96 8.44 +21.26%
25.57 12.21 -52.25% 7.09 8.65 +22.00%
25.33 11.98 -52.70% 6.94 8.40 +21.04%
25.56 12.20 -52.27% 6.77 8.16 +20.53%
25.29 11.69 -53.78% 6.71 7.80 +16.24%
25.19 11.61 -53.91% 6.71 7.91 +17.88%
25.07 11.63 -53.61% 7.02 8.31 +18.38%
25.14 11.60 -53.86% 6.88 7.94 +15.41%
ham mean and sdev for all runs
25.35 11.93 -52.94% 6.95 8.33 +19.86%
spam mean spam sdev
80.93 90.31 +11.59% 7.72 7.59 -1.68%
81.17 90.59 +11.61% 7.73 7.68 -0.65%
81.36 90.72 +11.50% 7.52 7.40 -1.60%
81.51 90.91 +11.53% 7.40 7.16 -3.24%
81.02 90.54 +11.75% 7.19 6.93 -3.62%
81.26 90.68 +11.59% 7.41 7.23 -2.43%
81.03 90.49 +11.67% 7.52 7.25 -3.59%
81.08 90.61 +11.75% 7.48 7.29 -2.54%
81.47 90.93 +11.61% 7.54 7.21 -4.38%
80.93 90.40 +11.70% 7.95 7.80 -1.89%
spam mean and sdev for all runs
81.18 90.62 +11.63% 7.55 7.36 -2.52%
ham/spam mean difference: 55.83 78.69 +22.86
Brad Clements, bkc@murkworks.com (315)268-1000
http://www.murkworks.com (315)268-9812 Fax
AOL-IM: BKClements