Mailman 3 April 2003 - SpamBayes

RE: [Spambayes] big imapfilter.py problem
by Meyer, Tony April 28, 2003

April 28, 2003

> What's a message database, and where do I find it? > OK, I presume spambayes.messageinfo.db is it. Sorry, yes. I've also confirmed with Tim that this is the cause. > Okay, training worked, but: > File "./spambayes/classifier.py", line 301, in probability > assert hamcount <= nham > AssertionError > % > What's the problem now? Hmm - interesting. This is the same problem as Olly was having. This means that the ham/spam count in the spambayes database (hammie.db, probably) is incorrect. My message to him had the two solutions. It's pushing the realms of coincidence a little to say that this has nothing to do with imapfilter. I'll try and see what's causing the count to go wrong. > File "imapfilter.py", line 372, in __cmp__ > return cmp(self.name, obj.name) > AttributeError: 'str' object has no attribute 'name'% I should be able to find this easily enough. Odd that it's not always coming up. > File "./spambayes/classifier.py", line 408, in _remove_msg > raise ValueError("spam count would go negative!") > ValueError: spam count would go negative! I suspect this is related to the first one. If I had to guess without looking at the code, I'd say that the untraining code is being activated when it shouldn't be. This would cause this one and the ham/spam count to be out. > different-every-time-ly y'rs. And just when I thought maybe things were going right... =Tony Meyer

2 1

Re: [Spambayes] imapfilter repeat proble
by Tim Stone - Four Stones Expressions April 28, 2003

April 28, 2003

4/28/2003 9:46:45 AM, Oliver Maunder <lists(a)olivermaunder.co.uk wrote: >Just been running imapfilter with the -l flag to get it to periodically >classify my inbox. The second round of classifying caused an error: > > "imaplib.error: command LOGIN illegal in state LOGOUT" > >Clearly, logging out after the initial round of classifying has left the >imap object in the LOGOUT state, and it doesn't want you to log in when >it's in that state. Personally I think that's probably an error in >imaplib (what else are you going to do with a logged out object other >than log in again?) . > >In the meantime there's a simple fix - move the line where the >IMAPSession is created inside the while loop, so a new IMAPSession is >created on each pass. >e.g. - line 633, imapfilter.py > while True: > imap = IMAPSession(server, port, imapDebug, doExpunge) > imap.login(username, pwd) > ... > >I still don't know much about python. Will the old IMAPSession object >will be cleaned up properly if I do this? It should be... I'll make that change. It makes sense anyway, because we shouldn't keep a socket tied up during a sleep time. > >Olly > > > >_______________________________________________ >Spambayes mailing list >Spambayes(a)python.org >http://mail.python.org/mailman/listinfo/spambayes > > c'est moi - TimS http://www.fourstonesExpressions.com http://wecanstopspam.org There are 10 kinds of people in the world: those who understand binary, and those who don't.

1 0

imapfilter repeat problem
by Oliver Maunder April 28, 2003

April 28, 2003

Just been running imapfilter with the -l flag to get it to periodically classify my inbox. The second round of classifying caused an error: "imaplib.error: command LOGIN illegal in state LOGOUT" Clearly, logging out after the initial round of classifying has left the imap object in the LOGOUT state, and it doesn't want you to log in when it's in that state. Personally I think that's probably an error in imaplib (what else are you going to do with a logged out object other than log in again?) . In the meantime there's a simple fix - move the line where the IMAPSession is created inside the while loop, so a new IMAPSession is created on each pass. e.g. - line 633, imapfilter.py while True: imap = IMAPSession(server, port, imapDebug, doExpunge) imap.login(username, pwd) ... I still don't know much about python. Will the old IMAPSession object will be cleaned up properly if I do this? Olly

1 0

[ spambayes-Bugs-728886 ] In the pop3 UI not able to pass more than 1 server
by SourceForge.net April 28, 2003

April 28, 2003

Bugs item #728886, was opened at 2003-04-28 12:38 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=498103&aid=728886&group_i… Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Remi Ricard (papadoc) Assigned to: Nobody/Anonymous (nobody) Summary: In the pop3 UI not able to pass more than 1 server Initial Comment: Hi, Using the Web UI to set the options for pop3proxy does not work. For Servers: I pass the string "mail.isp1.ca,pop.isp2.ca,pop.isp3.net" (without the " " ) In the file UserInterface.py before line 576 value = mail.isp1.ca,pop.isp2.ca,pop.isp3.net but after line 577 value = (mail.isp1.ca,pop.isp2.ca,pop.isp3.net,) Then you get this error message when you try to save # 'mail.gmc.isp1.ca,pop.isp2.ca,pop.isp3.net' is not a value valid for [POP3 Proxy Options] Servers # '6110,6111, 6112' is not a value valid for [POP3 Proxy Options] Ports Remi papaDoc(a)videotron.ca ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=498103&aid=728886&group_i…

1 0

RE: [Spambayes] big imapfilter.py problem
by Meyer, Tony April 28, 2003

April 28, 2003

> > File "./spambayes/classifier.py", line 301, in probability > > assert hamcount <= nham > > AssertionError > > What's the problem now? (Excuse the verbosity of the answer). I've established what the problem here is, I believe. The RFC says that a message's UID must be unique within the mailbox. I was not thinking clearly enough when I read that to realise that in the context of the RFC, a mailbox is a folder, not the whole collection of folders. This means that to get a stable, unique identifier for a message isn't possible - an unstable unique identifier could be obtained by combining the UID and the folder's UID validity value, but that isn't guaranteed to stay the same over different sessions. This explains why not everyone sees this. One of the servers I use has unique numbers for any message (judging from what I've seen) - I'm not sure about the other, but it might also - it's up to the server to decide how the UIDs are allocated. On the other hand, if you try to untrain the wrong message, you'll get lots of ham/spam count errors. So I'm going to give up using the UID (in any form) as an identifier for messages, and do what all the other spambayes apps (bar Outlook) do and add my own. I'll store this with the message whenever it's saved. This will mean things will be a little slower (have to search for messages with a header with a certain value, instead of for a message with a particular uid), but slow and working is better than fast and not. I'll have to rework a reasonable chunk of imapfilter to do this. It will also mean that the message info databases (spambayes.messageinfo.db, probably) will be invalid (the ids will be changing), although they should still work. Any training done with imapfilter is now suspect, so I wouldn't advise keeping hold of those db's (hammie.db etc). I'll try and do this ASAP, but real life is keeping me a bit busy over the next couple of days. In other notes, I've found the problem that was causing the __cmp__ error and fixed it (I committed another error at the same time, but I'll check in a fix for that soon). I've also found a place where things are much slower than they need to be, so performance gains are still easily possible. =Tony Meyer

3 2

RE: [Spambayes] gratuitous changes? + bugs
by Meyer, Tony April 28, 2003

April 28, 2003

> I did a cvs update today, and I clearly shouldn't have. My > programs based on spambayes ceased working for a variety of reasons. Note that I have endeavoured to post lots of notification about the Options [proposed] changes to the list, and the convert script *should* be able to convert any old config files. I've tried to include as much backward compatibility code as possible, but there are always going to be issues when changing part of the code that's as widely used as Options.py. Getting buggy code is part of the joy of cvs <wink>; along with reporting the bug and having it get fixed, of course. > 1. OptionClass.mergefiles was renamed to OptionClass.merge_files. > Gratuitous change? Or is there some deeper reason behind > this. The change was not mentioned in the cvs log, by the way. What do you have that calls mergefiles? None of the scripts in cvs do (well, a search for "mergefiles" doesn't bring anything up). It was renamed because mergefiles is not well named - it should be MergeFiles, mergeFiles or merge_files. It was renamed now because there are lots of other changes happening with Options.py, and it makes sense to fix everything at once. I suppose it should have been mentioned in the cvs log, sorry, but since I couldn't find anything that called it, I didn't rate it as worthy of specific inclusion. > 2. Boolean values in my .ini file (or at least the verbose flag) can't > be parsed anymore. The reason is, the check is_boolen is incorrect > for Python 2.3. It checks that the allowed_values is a bool, but in > actual fact it's a tuple of bools. See the diff. "globals":"verbose" should not be a tuple of bools, it should be a bool - it makes no sense for it to be a tuple. Options.py has the default as False - a single bool. What does your configuration file have? This could definitely be a bug (although it's not that likely that it's just for 2.3) - but the bug isn't in the check, it's in reading the value. If you can let me know the config line that's causing the problem, I'll try and figure a fix ASAP. > 3. My setting of basic_header_skip and safe_headers didn't work > anymore. The header names now have to be separated with a single > space. Why? I propose the fix included in the diff. Sorry, my mistake; you're absolutely correct. I'll check in the fix for this in a moment. =Tony Meyer

2 1

RE: [Spambayes] big imapfilter.py problem
by Meyer, Tony April 28, 2003

April 28, 2003

> What's happening here is in: > > File "spambayes/message.py", line 118, in _getState > (msg.c, msg.t) = self.db[msg.getId()] > > self.db[msg.getID()] > > is a email.Message object [...] TimS might have a better idea, but I *think* that this is a result of the message db being fixed ;) The db used to store message objects, and now stores a tuple of the classification and training information. So self.db[msg.getID()] should return a tuple, *not* an email object. Delete (or rename) your message database and this should work. (This won't effect your classification database, just the memory of which messages have been classified/trained, which was inaccurate anyway). Apologies for this - we should have realised that old (message) databases would become invalid and need to be removed and posted a message to that effect. =Tony Meyer

2 3

gratuitous changes? + bugs
by Meyer, Tony April 28, 2003

April 28, 2003

[Tony] > What do you have that calls mergefiles? None of the scripts in cvs do > (well, a search for "mergefiles" doesn't bring anything up). Well, so much for relying on XP's search. HammieFilter does call mergefiles. I'm about to check in a fix for this. Apologies that I didn't find it earlier. (This is an ugly way to do things, though; HammieFilter shouldn't be calling mergefiles, that's what the Options modules is for...) =Tony Meyer

1 0

big imapfilter.py problem
by David Abrahams April 27, 2003

April 27, 2003

%python imapfilter.py -t -v Loading database hammie.db... Loading state from hammie.db database hammie.db is a new database Done. Training Training ham folder HamBox Traceback (most recent call last): File "imapfilter.py", line 661, in ? run() File "imapfilter.py", line 647, in run imap_filter.Train() File "imapfilter.py", line 488, in Train num_ham_trained = folder.Train(self.classifier, False) File "imapfilter.py", line 431, in Train for msg in self: File "imapfilter.py", line 383, in __iter__ yield self[key] File "imapfilter.py", line 423, in __getitem__ msg.setId(key) File "spambayes/message.py", line 173, in setId msginfoDB._getState(self) File "spambayes/message.py", line 118, in _getState (msg.c, msg.t) = self.db[msg.getId()] File "/usr/home/dave/src/email-2.5/email/Message.py", line 283, in __getitem__ return self.get(name) File "/usr/home/dave/src/email-2.5/email/Message.py", line 349, in get name = name.lower() AttributeError: 'int' object has no attribute 'lower' What's happening here is in: File "spambayes/message.py", line 118, in _getState (msg.c, msg.t) = self.db[msg.getId()] self.db[msg.getID()] is a email.Message object which doesn't support indexing by ints, and the construct a, b = expr basically evaluates expr and then tries to treat it as a sequence and iterate over its elements, grabbing the first two and sticking them in a and b. I have no idea how or why this works for anyone else; it seems like a straight-up bug to me, but maybe something went wrong long before that I'm unable to diagnose. -- Dave Abrahams Boost Consulting www.boost-consulting.com

1 0

RE: [Spambayes] Classifier Assertion error
by Meyer, Tony April 27, 2003

April 27, 2003

> This is happening on a brand new DB, training with imapfilter. It's > happened on two separate PCs (home and work), but with similar setup > (Win XP, Python 2.2, pybsddb) and the same training data. > I'll give it a go under Linux when I get round to rebooting. Hopefully someone else will chime in here - I haven't really paid much attention to this bug (other that noting that it needs to be fixed before beta1) since I've never come across it myself. > > (b) Use dbimpexp.py to convert your database to a text > > file. Change the ham and spam counts at the top to something > > like the numbers they should be > Tried that. It works. Not sure yet what further training wiil do to > those counts though. Obviously it's not something you want to do all the time, anyway. If this happens regularly for you, it would be great if you were able to track down where the problem is occurring - I think that that is part of the problem - we're not sure what is causing the problem. A question for TimP (or any of the other stats people): if the ham/spam count does get lost, would setting them to the highest number of occurrences in the db screw things up? i.e. if my most spammy word appeared in 423 emails, and my most hammy word appeared in 233 emails, could I then set hamcount to 233 and spamcount to 423? =Tony Meyer

1 0