Mailman 3 October 2002 - SpamBayes

Advantages of S?
by Gary Robinson April 26, 2003

April 26, 2003

At present, it appears that S may perform better than relatively untweaked graham (as I guess bogofilter is using), but about the same as tweaked graham as spambayes is using, but has the advantage of showing a spread of results. As I see it there are two possible advantages to the spread: 1) emails in the middle could be classified as "you should manually check these", or 2) the user could adjust for the level of sensitivity he wants (similar to what happens now in Entourage which is otherwise brain-dead as a spam filter). (1) is a attractive but one thing worries me: it may give people a false sense of security -- they may think they only have to look at the middle ones, but occasionally even extreme cases of seeming to be spam might be legit. On the other hand, on those rare occasions, the sender can always try again if he doesn't get a response! ;) Just a couple of thoughts. --Gary -- Gary Robinson CEO Transpose, LLC grobinson(a)transpose.com 207-942-3463 http://www.emergentmusic.com http://radio.weblogs.com/0101454

8 19

Email client integration -- what's needed?
by Robin Munn Nov. 5, 2002

Nov. 5, 2002

Hi all, I just joined the spambayes mailing list a couple of days ago and have been trying to skim through the archives. It looks like a lot of time is being spent on algorithm refining and not as much time on email client integration or end-user documentation. I'm a Python programmer who's currently unemployed (though the "unemployed" part of that might be changing soon, I hope), so at the moment I have quite a bit of time to contribute. I want to see a project that's useable not just by techies but also by computer-illiterate end-users. So I'm offering my services to write client integration code or documentation. I don't want to duplicate work that's already being done by someone else, so can anyone tell me what's already being worked on -- and what the most urgent TODO items are? Thanks. -- Robin Munn <rmunn(a)pobox.com> http://www.rmunn.com/ PGP key ID: 0x6AFB6838 50FF 2478 CFFB 081A 8338 54F7 845D ACFD 6AFB 6838

12 19

Re: [Spambayes] Email client integration -- what's needed?
by T. Alexander Popiel Nov. 5, 2002

Nov. 5, 2002

In message: <SRMJLJVPA8WQQ43874XEC51RQ1USNZX.3dc1bcb9@riven> Tim Stone Four Stones Forum <tim(a)fourstonesforum.com> writes: >But can it be useful to the masses? The pop3proxy is the right way to go >in my opinion. You folks make me feel like such a fuddy-duddy, still using MH from a shell account with the mailboxes fetched through the filesystem, instead of through some network mailbox protocol... Heck, I don't even have software to access a POP mailbox installed... I guess that raises the question: what is our target audience, and how strictly do we want to cater to them? Do we want to offer support for processing in direct-delivery situations, even though it's only old-school fuddy-duddies like myself who use them, anymore? - Alex

9 20

Database reduction
by Tim Peters Nov. 4, 2002

Nov. 4, 2002

There's a semi-standard trick for database size reduction I haven't persued and don't intend to pursue. Those keenly interested in reducing database size may wish to pursue it. Currently, the classifier's wordinfo dict is indexed by strings S. There's no bound on how many unique strings may appear, and so also no bound on how large the database may grow. A cheesy but probably-effective trick is to pick an integer N for all time, and index a wordinfo structure by hash(S) % N instead of by S. Since strings are no longer stored: + Good: Space for storing strings isn't needed. + Bad: You can't get words out again (for, e.g., clue lists). Since hash(S) is a many-to-one mapping: + Bad: Words get combined more-than-less randomly. Since mod N is also a many-to-one mapping: + Bad: As above. + Good: N is a solid upper bound on the maximum number of wordinfo records, and so you can know exactly how big a classifier can get. + Good: You could drop the dict and use hash(S)%N to index a contiguous structure directly, like an mmap'ed file (http://crm114.sf.net/ uses that specific trick after multiple layers of hashing, into distinct files of one-byte clamped ham and spam counts). Since database size would be bounded: + Good: There's less obvious need to prune the database over time (a main point of pruning is to reclaim space for words that aren't being used anymore -- or ever). + Bad: If the database is never pruned, it will adapt slower to changes in the nature of ham and spam. I suppose the scariest thing is combining words "at random". It's possible, e.g., that Python would get mapped to the same record as Viagra. And the smaller N is, the most certain "bad stuff like that" *will* happen. We won't know until someone tries it and measures results; my intuition <wink> is that unless you get silly with N, it won't hurt much, as most words are approximately worthless anyway. Think about what happens when N=1 for the other side of this coin.

6 11

Re: pop3proxy bug?
by Richie Hindle Nov. 1, 2002

Nov. 1, 2002

Hi Jeremy, Sorry for the delay - I've been away since Friday. > Did you ever test this code? Yes, of course I tested it - I've been using it to retrieve all my email since the day I wrote it! 8-) The problem is probably down to platform-dependent behaviour - I'm running on Windows 98 and it works like a charm for me. I'll give it a go on Linux over the next day or two and see what happens. > I changed the code to use the raw server socket and sendall() instead > of self.serverFile.write() and it worked. But I'm uneasy. That's a perfectly reasonable fix. Once I've reproduced the problem on Linux, I'll apply, test and commit that fix - thanks. -- Richie Hindle richie(a)entrian.com

2 3

Demo Outlook Plugin available
by Mark Hammond Nov. 1, 2002

Nov. 1, 2002

Hi all, I just released new win32all builds that contain support for Microsoft Outlook Extensions. If you install win32all-149, 150 or the most recent CVS snapshot build, you will find a file win32com\demos\outlookAddin.py - please see the comments in the file for information on how to install and test this plugin. Please let me know if you try it. Also, feel free to contact me if you need some help turning it into something useful. Mark.

3 4

non-ascii mail in hammiecli
by Toby Dickenson Nov. 1, 2002

Nov. 1, 2002

hammiecli is giving its input an xmlrpc binary wrapper, to avoid marshalling problems with non-ascii input. However hammiesrv wasnt doing the same for its output. diff attached

2 1

Re: [Spambayes] Email client integration -- what's needed?
by Tim Stone Four Stones Forum Oct. 31, 2002

Oct. 31, 2002

Van, you and Robin have started to take this project to the next level. Thanks. I'm anxious to see what comes next. That Spambayes is great at filtering is unquestionable. But can it be useful to the masses? The pop3proxy is the right way to go in my opinion. I use an SMTP proxy now, it's so easy to set up, and it just works. Let me volunteer my services as a tester. I would consider myself to be a completely typical email user. I use the Opera mailer on a windoze platform. I certainly have plenty of spam volume... I also have a linux box that makes a nice little playground. 10/31/2002 5:09:27 PM, "G. Armour Van Horn" <vanhorn(a)whidbey.com> wrote: >I'm really new to this as well, having only joined yesterday. (I did read Paul >Graham's piece a couple of months ago though.) I'm not a programmer, just a >stumbling system administrator, so my interests run in a different direction. > >I currently am running MailScanner with SpamAssassin on a low-volume mail >server. I'd really like to have the Bayesian filter on the server instead of >SpamAssassin (or in addition, I suppose). I'd want to have two mail addresses >(one for spam, one for ham) that users could return messages to. I know the >ultimate precision of the system comes from having individuals do this for their >own mailboxes, but I think the community of users on this server is small enough >that we would get 99% of the benefit. > >I could be wrong, but I think this would work for a lot of company servers, >probably for a lot of small ISPs as well, protecting thousands of users both >from Spam and the need to install tricky software on their machines. > >Van > >Robin Munn wrote: > >> Hi all, >> >> I just joined the spambayes mailing list a couple of days ago and have >> been trying to skim through the archives. It looks like a lot of time is >> being spent on algorithm refining and not as much time on email client >> integration or end-user documentation. I'm a Python programmer who's >> currently unemployed (though the "unemployed" part of that might be >> changing soon, I hope), so at the moment I have quite a bit of time to >> contribute. I want to see a project that's useable not just by techies >> but also by computer-illiterate end-users. So I'm offering my services >> to write client integration code or documentation. I don't want to >> duplicate work that's already being done by someone else, so can anyone >> tell me what's already being worked on -- and what the most urgent TODO >> items are? Thanks. >> >> -- >> Robin Munn <rmunn(a)pobox.com> >> http://www.rmunn.com/ >> PGP key ID: 0x6AFB6838 50FF 2478 CFFB 081A 8338 54F7 845D ACFD 6AFB 6838 >> >> ------------------------------------------------------------------------ >> Part 1.2Type: application/pgp-signature > >-- >---------------------------------------------------------- >Sign up now for Quotes of the Day, a handful of quotations >on a theme delivered every morning. >Enlightenment! Daily, for free! >mailto:twisted@whidbey.com?subject=Subscribe_QOTD > >For web hosting and maintenance, >visit Van's home page: http://www.domainvanhorn.com/van/ >---------------------------------------------------------- > > > >_______________________________________________ >Spambayes mailing list >Spambayes(a)python.org >http://mail.python.org/mailman/listinfo/spambayes > > > >

1 0

Re: [Spambayes] FW: [Spambayes-checkins] spambayes tokenizer.py, 1.57,1.58
by Tim Peters Oct. 31, 2002

Oct. 31, 2002

[Brad Clements] > I get valid messages with embedded images that contain cid: clues. > Hopefully my ham tokens will overpower ;-) We work on sets of words, so no matter how many src=cid: instances appear in a single msg, only one will count against it -- provided your training data is such that src=cid: is actually a spam clue! In your corpus, it may be a ham clue. Remember that there's no way to force the system to consider anything to be ham or spam -- it learns the correct classification, and how strong an indicator a thing is, purely from what you train it on. In that sense too Guido's characterization shouldn't be taken seriously: we're only letting the classifier see these things, not telling it what to think about them.

1 0

FW: [Spambayes-checkins] spambayes tokenizer.py,1.57,1.58
by Tim Peters Oct. 31, 2002

Oct. 31, 2002

FYI, for those not on the checkin list. -----Original Message----- From: spambayes-checkins-bounces(a)python.org [mailto:spambayes-checkins-bounces@python.org]On Behalf Of Tim Peters Sent: Thursday, October 31, 2002 1:43 AM To: spambayes-checkins(a)python.org Subject: [Spambayes-checkins] spambayes tokenizer.py,1.57,1.58 Update of /cvsroot/spambayes/spambayes In directory usw-pr-cvs1:/tmp/cvs-serv30231 Modified Files: tokenizer.py Log Message: A new mini-phase of body tokenization scours HTML for common virus clues, variations of <script </script <iframe </iframe src=cid: height=0 width=0 I'm seeing a lot of this in my personal email lately, and it usually ends up in my Unsure folder because the msgs have almost nothing in them except for a bit of triggering HTML. Adding this stuff almost always scores them as solid spam now, and had no effect on my c.l.py test (no change in FP or FN rates, insignificant improvement in Unsure rate): filename: cv tcap ham:spam: 20000:14000 20000:14000 fp total: 2 2 fp %: 0.01 0.01 fn total: 0 0 fn %: 0.00 0.00 unsure t: 97 96 unsure %: 0.29 0.28 real cost: $39.40 $39.20 best cost: $26.80 $26.80 h mean: 0.26 0.27 h sdev: 2.89 2.90 s mean: 99.94 99.94 s sdev: 1.44 1.44 mean diff: 99.68 99.67 k: 23.02 22.97 Index: tokenizer.py =================================================================== RCS file: /cvsroot/spambayes/spambayes/tokenizer.py,v retrieving revision 1.57 retrieving revision 1.58 diff -C2 -d -r1.57 -r1.58 *** tokenizer.py 29 Oct 2002 03:43:58 -0000 1.57 --- tokenizer.py 31 Oct 2002 06:42:48 -0000 1.58 *************** *** 948,951 **** --- 948,967 ---- return ''.join(new_text), clues + # Scan HTML for constructs often seen in viruses and worms. + # <script </script + # <iframe </iframe + # src=cid: + # height=0 width=0 + + virus_re = re.compile(r""" + < /? \s* (?: script | iframe) \b + | \b src= ['"]? cid: + | \b (?: height | width) = ['"]? 0 + """, re.VERBOSE) + + def find_html_virus_clues(text): + for bingo in virus_re.findall(text): + yield bingo + class Tokenizer: *************** *** 1219,1222 **** --- 1235,1241 ---- for t in tokens: yield t + + for t in find_html_virus_clues(text): + yield "virus:%s" % t # Remove HTML/XML tags. Also  . _______________________________________________ Spambayes-checkins mailing list Spambayes-checkins(a)python.org http://mail.python.org/mailman/listinfo/spambayes-checkins

3 3