Mailman 3 October 2004 - spambayes-dev

A query on Frequently Asked Question 4.6
by Harry Sigerson Oct. 30, 2004

Oct. 30, 2004

Dear SpamBayes, A query on FAQ 4.6 ================ 4.6 Do I need to keep spam after it has been trained? If so, for how long? Once a message has been [correctly] trained there is no need to keep it around. However, SpamBayes' accuracy is dependent upon having a "sufficient" sample from which to make its decisions. Therefore, most users retain a fair amount of spam in the event that they may wish to rebuild the corpus from scratch. Of course, this begs the question: "how much is enough?" That is where the "art" of SpamBayes meets the science. Some users keep as many as several thousand [recent] spam (as well as a similar number of ham). That is not to say that you won't have excellent results with a tenth (or less) of that number; since everyone's e-mail profile is different, the requirements for training are as well. ================ The FAQ item 4.6 above addressed something that I've wondered about since I got SB going around July this year, that is, "Where does all the 'saved' spam go?" <smile>. I do have a lot of spam now, 16,054 spams and 1,725 hams. The thing is where do I go to cut down on the amount of spam? I see the comment, "Warning: you have much more spam than ham - SpamBayes works best with approximately even numbers of ham and spam." in the 'Status and Configuration' section. I'll have another look and see if I can figure out where all the spam is located... ================ 4.3 How do I train SpamBayes (forward/bounce method)? Alternatively, when you receive an incorrectly classified message, you can forward it to the SMTP proxy for training. If the message should have been classified as spam, forward or bounce the message to spambayes_spam@localhost, and if the message should have been classified as ham, forward it to spambayes_ham@localhost. You can still review the training through the web interface, if you wish to do so. You should ensure that the "lookup message in cache" option is set to True/Yes before you use this. Note that some mail clients (particularly Outlook Express) do not forward all headers when you bounce, forward or redirect mail. We do not recommend using the SMTP proxy with these clients. =============== ...I saw from that item 4.3 that there are locations at... spambayes_spam@localhost ...and at... spambayes_ham@localhost ...are these locations on this my computer or are they located at my ISP's server? This is a standalone machine with a cable-modem broadband connection to NTL... I've just used XP's Search facility to see if, spambayes_spam@localhost , exists as a file on this machine it didn't find such and I didn't expect that it would. Are these thousands of spam actually located on my ISP's server, that is on NTL's server? If so how do I go about accessing this that I might reduce the amount of spam to something near the same 'ham' figure of 1,725? Excuse the long explanation. Regards, Harry Sigerson.

1 0

need help
by mwm＠aaahawk.com Oct. 27, 2004

Oct. 27, 2004

I down loaded spambayes but wasn't savvy enough to install it properly. I unistalled it. Now Outlook Express "can't find my smtp server". I can get onto the net ok and can use webmail from my ISP but always get the above error when trying to receive or send mail. Thanks

1 0

State of I18N
by Hernán Martínez Foffani Oct. 22, 2004

Oct. 22, 2004

I made some tests and managed to build a partially translated version of the plugin. Follows what I found so far. Please consider every paragraph I wrote ending with "What do you think?", "Is this the right approach?" or "Am I screwing up something?" kind of questions. GET THE USER LANGUAGE I'm using locale.getdefaultlocale() to get the user lang preference. I'll add the chance to let the user specify another language in his config file. A different approach would be to get Outlook's language-id version (how?) instead of the user's locale and convert it later to Python's RFC1766 format code. Languages are "stackable" so they can naturally fallback one over the other if their translations are missing or incomplete. PURE PYTHON LITERAL STRINGS By using gettext, I found that adding a small class and a couple of lines at the begining of manager.py:BayesManager.__init__() is enough to have the infrastructure for the python code part. I'll leave out of translation assert's and LogDebug comment strings. The rest would be tedious but easy to do. I'll follow Tony's advise and start the addin translation after sb_server but I want to define the translation tools first. Some things may require some rework though: - Multiline messages that use one print sentence per line. (Replace those by "xxx\r\n" "yyy\r\n" "zzz\r\n"?) - Those that mix literal strings with parameters concatenating them instead of using placeholders. - The special case of mix literals with HTML. (see addin.py:ShowClues) - There are literals words around that might be used for program flow (eg: "ham","spam",etc.) Not sure about it. Anyway, no big surprises. gettext seems good for several reasons: - python officially blessed it, - python dist has (almost) all the tools a translator needs, - minimizes changes on source code, - includes fallbacks to language/country and - I like it. DIRECTORIES The gettext library uses {SomeDir}/langcode/LC_MESSAGES/message_file layout. In the sources I'm working on I made {SomeDir} as "Languages" parallel to Outlook2000, pspam and scripts directories. Doing so I can place all the translated messages files in one directory and manager.py&co can reach them at ../Languages. Is it too hard to test this layout for a frozen distribution? DIALOGS These were tricky. Binary dist SB imports them directly while the source addin compiles them first. I don't think its necessary to reproduce the same thing for foreign dialog.rc's. A translator can run rc2py at the command prompt to create the corresponding .py file. To imitate the fallback behaviour for dialogs I had to play a bit with sys.path. Setting a language, besides initializing gettext, appends __file__/../Languages/langcode/DIALOGS and __file__/../Languages/langcode[0:2]/DIALOGS to sys.path. (Actually it's not __file__ but this_filename and it's not [0:2] but the first "_" split.) Say the locale "es_ES", then SB will try to import "i18n_dialog.py" from "Languages/es_ES/DIALOGS" then from "Languages/es/DIALOGS". If it fails it fallbacks to the current code. GETTEXTED DIALOGS I mentioned before that I would like to make rcparser.py gettext aware so a translator could work out a solution without having to edit the resource dialogs. The problem I found is that the output of the parser is a dict that includes string literals subject to be translated (like caption or labels) and literals that do not (like font names.) Once in the dict there's no way to differentiate between them. As I expect that changes in the dict would imply changes all over SB, I thought of a partial solution (call it a hack) that may work. AFAICT, SB does not use the output of rcparse directly but through a rc2py previous process. So how about a subclass of str that override __repr__ ? Then, rcparse only needs to create instances of this class for labels, captions, etc., not for font names (or every literal can be of this new type and an attribute flag can drive __repr__ accordingly.) Later, rc2py and imports of the dialogs .py would do the right thing. Something like this: class gt_str(str): def __repr__(self): return "_(" + super(gt_str, self).__repr__() + ")" >>> a="a" >>> b=gt_str("a") >>> a==b True >>> a 'a' >>> b _('a') >>> I said it's a partial solution because the dict that rcparse generates does not equal (items of different types) the one obtained by importing the dialog. Does it matter? (It's also "partial" because I haven't tested it yet. heh..) TRANSLATOR TOOLS With this approach a translator would need: - Outlook with SB (binary dist is enough) - python (for the gettext tools) - the rcparse/rc2py tools also highly recommended: - a free resource editor and optionally: - SB source distro (i think it's no needed, really) To translate: - the "all_the_messages_in_SB" file (I can provide it for each new version of SB) - the dialogs.rc file for each new version of SB. Still, I have to search for a tool or procedure to let a translator knows which messages are not translated yet. If you know of a smart .po (gettext messages files) comparition tools, tell me please. That's all for the time being. Soon, I'm going to polish the changes I made and load the corresponding patches to sf.net for your review. Regards, -Hernán. PS: Please, bare my english.

6 9

Windows Installer not offering to add startup icon
by Tony Meyer Oct. 22, 2004

Oct. 22, 2004

It appears that our installers aren't offering to create a startup icon for sb_server users as it should be. I've checked 1.0a9, 1.0rc1, 1.0rc2 and 1.0 and they all skip it. (I'm fairly impressed that sb_server users are clever enough to do it themselves!). The problem is the "Check: InstallingProxy" line in the Inno script. Although the selection code has been run by the time the tasks are offered, it still has the default (False) value. I've played around with the code, but can't figure a way around this (although my Pascal is extremely rusty). We can fix it by removing that Check. Outlook users don't get that page anyway, so they won't see the option (this is what happens with the desktop icon). However, since we want it checked by default, it will appear in the text box of additional tasks, even though it doesn't happen. Does this matter? I'm doubtful that anyone reads that box anyway, so my vote is no. If no-one complains, then I'll check the change in. =Tony Meyer P.s. If anyone wants to do a sanity check for me and run any of the installers and see if it gets offered, that would be great ;)

1 0

RE: [spambayes-dev] State of I18N
by Tony Meyer Oct. 21, 2004

Oct. 21, 2004

> I'm using locale.getdefaultlocale() to get the user lang > preference. I'll add the chance to let the user specify > another language in his config file. > > A different approach would be to get Outlook's language-id > version (how?) instead of the user's locale and convert it > later to Python's RFC1766 format code. I'd stick with the former, so that this naturally carries past the Outlook plug-in into the other scripts. Something like a [globals] language option, that allows multiple string values (a list of preferred languages in order). I can add that if you like. I would probably have the default (in Options.py) be "English", and then have a page on the configuration wizard(s) that lets you select the preferred language(s), and have that default to locale.getdefaultlocale() (or maybe, for Outlook, the Outlook language). > I'll leave out of translation assert's and LogDebug comment > strings. Yes, there isn't much point in translating those. In fact, it could make things harder, since I wouldn't want to have to try to figure out someone's problem given a log in (eg) Spanish. > Some things may require some rework though: > - Multiline messages that use one print sentence per line. > (Replace those by "xxx\r\n" "yyy\r\n" "zzz\r\n"?) Is this to make the translating easier so you have more context, or for some other reason? > - Those that mix literal strings with parameters concatenating > them instead of using placeholders. You mean "string" + var + "string" instead of "string %s string" % (var,)? I'd much rather stick with the latter, if possible. > - The special case of mix literals with HTML. > (see addin.py:ShowClues) This is the only case of that (other than the whole of the web interface, which is in ui.html), IIRC. We can probably figure out something to do there. > - There are literals words around that might be used for > program flow (eg: "ham","spam",etc.) Not sure about it. Ignore those, and I'll check them in the patch. There shouldn't be - for example, "ham", "spam" and "unsure" are all configurable options, so they ought to be referred to via the options class. (This might not always be the case with the Outlook plug-in, but I can fix that if necessary). > DIRECTORIES > > The gettext library uses {SomeDir}/langcode/LC_MESSAGES/message_file > layout. In the sources I'm working on I made {SomeDir} as > "Languages" parallel to Outlook2000, pspam and scripts directories. "languages" in that directory sounds fine to me for the source. > Doing so I can place all the translated messages files in one > directory and manager.py&co can reach them at ../Languages. > > Is it too hard to test this layout for a frozen distribution? We can setup the binary however we like. At the moment, in the application directory there is a "bin" directory and a "lib" directory. I suppose "languages" ought to go in the top level or in "lib" - the top level would match the source, so it might as well be that, I guess. > To imitate the fallback behaviour for dialogs I had to play > a bit with sys.path. Setting a language, besides > initializing gettext, appends > __file__/../Languages/langcode/DIALOGS and > __file__/../Languages/langcode[0:2]/DIALOGS to sys.path. > (Actually it's not __file__ but this_filename and it's not > [0:2] but the first "_" split.) > > Say the locale "es_ES", then SB will try to import > "i18n_dialog.py" from "Languages/es_ES/DIALOGS" then from > "Languages/es/DIALOGS". If it fails it fallbacks to the current code. Rather than using __file__ (which is what is in this_filename?), use manager.application_directory (or self.application_directory if this is in the Manager class). (But I can make this change to any patch, anyway). > GETTEXTED DIALOGS [...] This sounds workable to me, but I'd have to see it in action to really get an idea of what is happening (I'm only vaguely familiar with rcparse anyway). =Tony Meyer

2 1

RE: [spambayes-dev] Any updated information for the planstomakebetterstatistics?
by Tony Meyer Oct. 20, 2004

Oct. 20, 2004

> Is there any way you can add a decimal point to the stat > percentages? = ) Done. The oastats.Stats().GetStats() function now takes a "decimal_places" argument that does what you would expect. I've set it to default to 1, but you could modify that for your local copy if you wanted. =Tony Meyer

1 0

RE: [spambayes-dev] Any updated information for the planstomakebetterstatistics?
by Tony Meyer Oct. 19, 2004

Oct. 19, 2004

> I have been testing this and I am catching an error. > It appears that it is not calculating the unsure > percentage from session to session, even though the > number of unsure messages is shown. Thanks; fixed. =Tony Meyer

2 1

RE: [spambayes-dev] Any updated information for the plans to makebetter statistics?
by Tony Meyer Oct. 17, 2004

Oct. 17, 2004

> Any updated information for the plans to make better > statistics (in terms of persistence, presentation, and > what's provided) especially for the Outlook addin? I'm planning on adding persistence (like sb_server/sb_imapfilter already have) in the near future (it would certainly make it into 1.1a1); I've done some of the coding already. Presentation is more tricky - I presume that what would be nice are some graphs, but I don't have the skills to draw a dynamic graph in an Outlook dialog (if anyone provides example code to do this, then I'd happily tie in the SpamBayes stuff). > Really, all I'm looking for is a persistent stat area > (much like the current per session one) but includes > classification accuracy percentage which is calculated > from the total messages processed and classification > errors. I'm presuming that the errors would be the > messages that were manually classified as good or spam > and/or the false positives and false negatives? I'm not sure about adding an "accuracy percentage", since it's hard to define (what do you do with unsures?), and so might mislead people, since there presumably wouldn't be room to explain it in the dialog. You're welcome to try and convince me otherwise, of course; it would certainly be an easy addition. However, once the stats are persistent, it would be simple for you to calculate any such figure yourself, of course. Would that suffice? > This way I can better test this wonderful tool and provide > much more concrete test results when using the Outlook > addin. In terms of the spam filtering accuracy when using > different training methods and/or options for the SpamBayes > engine. If you want to do serious testing, then you'd be better off using the testing scripts that are in the source archive (timcv.py and incremental.py). These don't rely on you consistently doing the same thing, and you can easily test multiple options/regimes over the same mail set. There's a bit of a learning curve involved, but it's not too bad. =Tony Meyer --- Please always include the list (spambayes(a)python.org) in your replies (reply-all), and please don't send me personal mail about SpamBayes. This way, you get everyone's help, and avoid a lack of replies when I'm busy.

2 1

RE: [spambayes-dev] Any updated information for the planstomakebetter statistics?
by Tony Meyer Oct. 17, 2004

Oct. 17, 2004

> Don't know if it's related, but I don't think anything else > has changed. I updated from CVS to get the stats changes and > now I'm getting an exception from the addin. Here's the > exception message: [...] > exceptions.AttributeError: '<win32com.client.COMEventClass instance at > 0x122933000>' object has no attribute 'name' Sorry, this was an unrelated change that I absent-mindedly checked in at the same time. (I copied over the stats stuff from the checkout that I use day-to-day to my main working copy to avoid checking in the Outlook tte stuff I'm playing around with - but that copy already had that change, which I forgot that I dumped after I figured it didn't work). Anyway, I've backed that bit out and it should work properly now. Apologies & thanks for pointing it out :) =Tony Meyer

2 2

RE: [spambayes-dev] RE: [Spambayes] Need assistance. Trying to make anOutlook binaryinstall from source.
by Tony Meyer Oct. 17, 2004

Oct. 17, 2004

> Since we're not supporting this anymore, is there any reason > not to remove the files in Outlook2000/installer from latest CVS? Good point; done. =Tony Meyer

1 0