[Spambayes] A Tim-Combining run
T. Alexander Popiel
popiel@wolfskeep.com
Thu, 10 Oct 2002 11:58:27 -0700
All of this is on my website at:
http://www.wolfskeep.com/~popiel/spambayes/timcomb
Tim combining looks interesting, but I can't really tell if it's
a win or a lose. The fp/fn rates only changed by a message or two.
On the other hand, Tim combining shoved one of my ham scores into
the 80s, and some of my spam scores into the single digits. Ouch.
For this one, I haven't done the full ratio analysis... I only did
the 1:1 case with 10 sets of 200 each. Like Brad, I ran it once to
find out what the cutoffs should be, then ran it again to utilize
those cutoffs. Default combining is cv1, Tim combining is cv2.
The one thing of note is that I seem to have surprisingly low cutoff
values (0.42 and 0.38).
Default combining, spam_cutoff 0.42:
-> <stat> Ham scores for all runs: 2000 items; mean 18.07; sdev 7.22
-> <stat> min 0.763677; median 17.9834; max 63.3652
* = 3 items
0 2 *
1 20 *******
2 14 *****
3 26 *********
4 33 ***********
5 21 *******
6 27 *********
7 29 **********
8 26 *********
9 25 *********
10 41 **************
11 56 *******************
12 53 ******************
13 88 ******************************
14 115 ***************************************
15 134 *********************************************
16 153 ***************************************************
17 142 ************************************************
18 130 ********************************************
19 159 *****************************************************
20 129 *******************************************
21 111 *************************************
22 87 *****************************
23 74 *************************
24 59 ********************
25 55 *******************
26 40 **************
27 28 **********
28 18 ******
29 17 ******
30 19 *******
31 8 ***
32 2 *
33 10 ****
34 8 ***
35 7 ***
36 12 ****
37 3 *
38 2 *
39 1 *
40 0
41 1 *
42 1 *
43 0
44 1 *
45 0
46 1 *
47 0
48 3 *
49 0
50 2 *
51 0
52 1 *
53 0
54 2 *
55 0
56 0
57 0
58 0
59 2 *
60 0
61 1 *
62 0
63 1 *
64 0
65 0
66 0
67 0
68 0
69 0
70 0
71 0
72 0
73 0
74 0
75 0
76 0
77 0
78 0
79 0
80 0
81 0
82 0
83 0
84 0
85 0
86 0
87 0
88 0
89 0
90 0
91 0
92 0
93 0
94 0
95 0
96 0
97 0
98 0
99 0
-> <stat> Spam scores for all runs: 2000 items; mean 75.97; sdev 9.00
-> <stat> min 19.5284; median 77.2341; max 98.0328
* = 2 items
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 0
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 1 *
20 0
21 0
22 1 *
23 0
24 1 *
25 0
26 0
27 0
28 0
29 1 *
30 1 *
31 0
32 0
33 0
34 1 *
35 0
36 0
37 0
38 0
39 0
40 0
41 0
42 1 *
43 0
44 2 *
45 4 **
46 4 **
47 4 **
48 2 *
49 3 **
50 7 ****
51 3 **
52 4 **
53 1 *
54 6 ***
55 12 ******
56 11 ******
57 12 ******
58 7 ****
59 17 *********
60 19 **********
61 21 ***********
62 17 *********
63 17 *********
64 31 ****************
65 31 ****************
66 44 **********************
67 32 ****************
68 53 ***************************
69 44 **********************
70 51 **************************
71 63 ********************************
72 76 **************************************
73 95 ************************************************
74 88 ********************************************
75 90 *********************************************
76 97 *************************************************
77 107 ******************************************************
78 106 *****************************************************
79 110 *******************************************************
80 110 *******************************************************
81 90 *********************************************
82 87 ********************************************
83 90 *********************************************
84 73 *************************************
85 63 ********************************
86 42 *********************
87 56 ****************************
88 26 *************
89 18 *********
90 13 *******
91 11 ******
92 8 ****
93 2 *
94 1 *
95 1 *
96 5 ***
97 3 **
98 3 **
99 0
-> best cutoff for all runs: 0.42
-> with weighted total 1*15 fp + 6 fn = 21
-> fp rate 0.75% fn rate 0.3%
-> matched at 0.43 with 14 fp & 7 fn; fp rate 0.7%; fn rate 0.35%
-> matched at 0.44 with 14 fp & 7 fn; fp rate 0.7%; fn rate 0.35%
With Tim combining, spam_cutoff 0.38:
-> <stat> Ham scores for all runs: 2000 items; mean 8.43; sdev 6.15
-> <stat> min 0.66161; median 7.56964; max 81.0785
* = 4 items
0 14 ****
1 95 ************************
2 115 *****************************
3 126 ********************************
4 169 *******************************************
5 189 ************************************************
6 188 ***********************************************
7 186 ***********************************************
8 176 ********************************************
9 178 *********************************************
10 128 ********************************
11 90 ***********************
12 64 ****************
13 76 *******************
14 45 ************
15 45 ************
16 21 ******
17 18 *****
18 14 ****
19 14 ****
20 3 *
21 5 **
22 7 **
23 5 **
24 3 *
25 1 *
26 3 *
27 2 *
28 1 *
29 3 *
30 0
31 0
32 0
33 0
34 0
35 1 *
36 0
37 1 *
38 0
39 1 *
40 0
41 0
42 0
43 1 *
44 0
45 0
46 1 *
47 2 *
48 0
49 0
50 1 *
51 0
52 1 *
53 0
54 0
55 1 *
56 0
57 0
58 1 *
59 0
60 0
61 1 *
62 0
63 0
64 0
65 0
66 0
67 3 *
68 0
69 0
70 0
71 0
72 0
73 0
74 0
75 0
76 0
77 0
78 0
79 0
80 0
81 1 *
82 0
83 0
84 0
85 0
86 0
87 0
88 0
89 0
90 0
91 0
92 0
93 0
94 0
95 0
96 0
97 0
98 0
99 0
-> <stat> Spam scores for all runs: 2000 items; mean 86.91; sdev 10.12
-> <stat> min 8.51412; median 89.773; max 98.1726
* = 3 items
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 1 *
9 1 *
10 0
11 0
12 0
13 0
14 1 *
15 1 *
16 0
17 1 *
18 0
19 0
20 0
21 1 *
22 0
23 0
24 0
25 0
26 1 *
27 0
28 0
29 0
30 0
31 0
32 0
33 1 *
34 0
35 0
36 0
37 0
38 3 *
39 1 *
40 1 *
41 0
42 1 *
43 1 *
44 2 *
45 4 **
46 0
47 2 *
48 1 *
49 2 *
50 5 **
51 2 *
52 2 *
53 1 *
54 2 *
55 2 *
56 1 *
57 0
58 5 **
59 8 ***
60 6 **
61 2 *
62 3 *
63 6 **
64 5 **
65 10 ****
66 10 ****
67 9 ***
68 12 ****
69 14 *****
70 11 ****
71 6 **
72 12 ****
73 14 *****
74 23 ********
75 18 ******
76 15 *****
77 20 *******
78 21 *******
79 30 **********
80 35 ************
81 39 *************
82 41 **************
83 66 **********************
84 72 ************************
85 75 *************************
86 98 *********************************
87 69 ***********************
88 118 ****************************************
89 112 **************************************
90 154 ****************************************************
91 151 ***************************************************
92 160 ******************************************************
93 140 ***********************************************
94 142 ************************************************
95 83 ****************************
96 91 *******************************
97 49 *****************
98 4 **
99 0
-> best cutoff for all runs: 0.38
-> with weighted total 1*14 fp + 8 fn = 22
-> fp rate 0.7% fn rate 0.4%
Finally, results.txt:
cv1s -> cv2s
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
false positive percentages
0.500 0.500 tied
0.500 0.500 tied
0.000 0.000 tied
1.000 1.000 tied
0.500 0.500 tied
2.000 1.500 won -25.00%
1.000 1.000 tied
0.500 0.500 tied
1.000 1.000 tied
0.500 0.500 tied
won 1 times
tied 9 times
lost 0 times
total unique fp went from 15 to 14 won -6.67%
mean fp % went from 0.75 to 0.7 won -6.67%
false negative percentages
0.500 0.500 tied
0.000 0.000 tied
1.000 1.500 lost +50.00%
0.500 0.500 tied
0.000 0.500 lost +(was 0)
0.500 0.500 tied
0.000 0.000 tied
0.000 0.000 tied
0.500 0.500 tied
0.000 0.000 tied
won 0 times
tied 8 times
lost 2 times
total unique fn went from 6 to 8 lost +33.33%
mean fn % went from 0.3 to 0.4 lost +33.33%
ham mean ham sdev
17.22 7.72 -55.17% 7.39 6.43 -12.99%
18.69 8.46 -54.74% 7.27 5.77 -20.63%
18.86 8.94 -52.60% 6.50 4.71 -27.54%
16.79 7.92 -52.83% 7.75 6.01 -22.45%
18.66 8.88 -52.41% 7.09 5.98 -15.66%
18.47 8.99 -51.33% 7.83 8.27 +5.62%
18.19 8.51 -53.22% 6.99 6.02 -13.88%
18.38 8.44 -54.08% 6.80 5.45 -19.85%
17.67 8.38 -52.57% 7.88 7.12 -9.64%
17.72 8.10 -54.29% 6.18 4.79 -22.49%
ham mean and sdev for all runs
18.07 8.43 -53.35% 7.22 6.15 -14.82%
spam mean spam sdev
75.58 86.54 +14.50% 9.15 10.45 +14.21%
76.81 87.80 +14.31% 8.53 8.21 -3.75%
74.95 85.60 +14.21% 9.44 12.09 +28.07%
76.18 87.24 +14.52% 8.64 9.83 +13.77%
76.55 87.63 +14.47% 8.84 9.54 +7.92%
76.08 86.83 +14.13% 8.69 10.19 +17.26%
75.61 86.38 +14.24% 9.72 11.14 +14.61%
76.51 87.65 +14.56% 8.30 8.75 +5.42%
75.92 86.79 +14.32% 9.62 11.13 +15.70%
75.52 86.64 +14.72% 8.76 8.95 +2.17%
spam mean and sdev for all runs
75.97 86.91 +14.40% 9.00 10.12 +12.44%
ham/spam mean difference: 57.90 78.48 +20.58
- Alex