Mailman 3 November 2006 - SpamBayes

problem
by kathi Nov. 6, 2006

Nov. 6, 2006

Hi, sorry but I am having a problem and unable to fix it. I accidently lost my junk suspects ( spam) on list and now when I get some emails they are there and then instantly gone....I can't find them anywhere. ( been waiting for a airline confirmation from Delta) 1. Where are those emails? (not in the regular spam or inbox) 2. how do I fix the problem? thanks so much Kathi Urie Shoreline Wa kathiu(a)verizon.net

1 0

Problem w/Spam Bayes
by L Tusly Nov. 6, 2006

Nov. 6, 2006

I have been using Spam Bayes successfully for 2-3 years with no complaint. Starting yesterday, when I go in my Inbox and hit Spam the first time, it works successfully; any times after that, it doesn't. Do I need to reload? Laurie

1 0

Windows compatibility - OCR [was: Unwanted stock solicitations]
by Vibe Grevsen Nov. 5, 2006

Nov. 5, 2006

Hi friends, OCR code's now been tweaked and tested to work in both WinXP and Win9x. This should work in unix as well. Here is a summary: 1. Put ocrad 0.16 in the path 2. Change the following in ImageStripper.py ocr = os.popen("ocrad -s %s -c %s -x %s < %s 2>ocrerr.txt" % (scale, charset, orf, pnmfile)) into this ocr_cmd = ur'ocrad -s %s -c %s "%s"' % (scale, charset, pnmfile) # os.popen3() returns [stdin, stdout, stderr] ocr = os.popen3( ocr_cmd )[1] 3. Change this if os.path.exists(program) and is_executable(program): into this if os.path.exists(program + ".exe") or ( os.path.exists(program) and is_executable(program) ): Because of the way the instruction is interpreted it does not produce fatal errors even if the file is not found. 4. Change this for line in open(orf): if line.startswith("lines"): nlines = int(line.split()[1]) if nlines: ctokens.add("image-text-lines:%d" % int(log2(nlines))) into this nlines = ctext.count('\n') if nlines: ctokens.add("image-text-lines:%d" % nlines ) 5. Finally I sugest you change the default scale from 1 to 2 like in this line scale = options["Tokenizer", "ocrad_scale"] or 2 Compile and enjoy. Happy coding :) Vibe

3 4

Unwanted stock solicitations
by Kasper Vibe Grevsen Nov. 5, 2006

Nov. 5, 2006

Hi friends, as posted elsewhere there is a NetPbm port for Windows. I think it would be beneficial for Windows users if it was possible to use that with SpamBayes instead of having to setup PIL/Python. I am trying to make ocrad run in combination with SpamBayes for Windows as well. First: The ocrad.exe in the download section is according to the version info really v. 0.14 and is exactly the same as the one in the windows installer package (and not v. 0.15 as tagged in the download area!). Secondly: Yes, the program runs from CMD and it actually produces very nice output, but the pnm-file you link to are apparently invalid. Third: You seem to have the scale parameter format wrong, that's why the output is so bad. Ok, here is how to test it: -Download IrfanView from http://www.irfanview.com/download_sites.htm -Download and open the pnm file from the link -Select Edit -> Copy then Edit -> Paste in the menu (to create a new, non-corrupted image) -Save as PGM (What I did, but other P*M formats should work too) -Now execute ocrad Here are two test runs. First one with incorrect parameter format as in your example. Second with correct scaling parameter: ----------------------------------------------------------------------------------------------------------------------------- >ocrad -4 Clipboard01.pgm ___AllE_llO_ ALL DA_ IRADER_ A_D |__E_IOR____ _omO_n_ __m_ _O_TWW__T_R__ W_Dl_AL. lh_ __o_k __mbol _wnw Wond__ _lo__ o 11 _ol_m_ _.TSl.TO_ _h_nO_ _P o o__ (_T TB_/_) w__k__ __o ___.ooo.ooo 00 (AOO_o_) _WWM p_ R___A___ BR_A_IWC w_w_ | _o__hw______ M_dl_ol _ol__lo__ |__ _PIWW_W__T_ _WWM_ |_ Ol_o__d _o o__o____ _ho_ |_ ho_ ______d |__o _h_ _o_mol __oO__ o_ __Oo_lo_lo__ __Oo_dl_O Po_|_|_ Rlm _o_lo__ dl___lb__lo_ _o_ |__ O_oO_|__o__ _obO_o_d_TM_ O_od___ ||__ D__lnO _h_ |___ _ mon_h_. _o__hw______ M_dl_ol ho_ h_ld dl_____lo__ Oo_h___d do_o o_d __Olo__d _h_ O_o_O___|__ ||____|_o o_ |__ _obO_o_d_TMl dloO_o__|_ _____m _o b_ _old _h_o_Oh |_____o_lo_oll_ bo__d dl___lb__lo_ _omOo_|__ |__o Po_|_|_ Rlm o__o_ Th_ |_|_lol |_______ ho_ b___ ___ _h_ _ol_ o_ _h_ O_od___ _o_ Oo____m___ _mOlo___ ____|_O O_oO_om_ ___h o_ Ooll__ _o____ lo_ol o_d __o__ |___| wo____ o_d ol_o |__|_d__ hlOh__ |___| wo_Wo____ R_o_o__ _|__d _o_ _h_ |_______ o_ _h_ _obO_o_d_TM_ O_od___ ho__ b___ _h_ __ho___d _o__ o_ ___ o_ _h_ O_od___ o_d lmm_dlo__ o_ _|__ _o__|_mo_lo_ o_ w_ll o_ |____o__d __lloblll__ o_ _h_ ____|__ l_w___ _k__oRIr_ h_w_ '_Ir_ _ow WOl_L) ADD lwl_ _EW lo _O_R RADAR A_D WAl_W IRADE o_ WO_DA_| DO_.| E_E_ BLl__| _w_w DOE__.| _LEEpl RPmo_al _ Dl__lalmP_ _ --------------------------------------------------------------------------------------------------------------------------------- > ocrad -s4 Clipboard01.pgm ?wwATTENTION ALL DAy.TRADERS AND INVESTORSwww . . . Company Name: 3auTww5_T5RPd M_ar__L, Ýh'_L Stock Symbol: 3w__q Monday Close: 0.11 Volume: 5,761,702 Change: uP 0.025 (27.78░_o) Market Cap: $33,000,000.00 (Approx) SWNM,PK RELEASES BREAKING NEWS ! . Sou_hwes_ern Medical Solu_ions, Inc, (PINKSHEETS_ SWNM), is pleased _o announce _ha_ i_ has en_e?ed in_o _he formal s_aes of nego_ia_ions regarding Pacific Rim na_ions dis_ribu_ion for i_s proprie_ary Labguard(TM) produc_ line, . . DurÝng _he las_ s mon_hs, Sou_hwes_ern Medical has held discussions, ga_hered da_a, and explored _he prospec_ive licensing of i_s Labguard(TM) diagnos_ic sys_em, _o be sold _hrough in_erna_ionally based dis_ribu_ion companies in_o Pacific Rim a_eas, The ini_ial in_eres_ has been fur _he sale of _he produc_ for governmen_ employee _es_ing programs such as police forces, local and s_a_e level workers, and also includes higher-level wor_orces, Reasons ci_ed for _he in_eres_ of _he Labguard(TM) produc_ have been _he enhanced ease of use of _he produc_ and immedia_e on-si_e confirma_ion as well as increased reliabili_y of _he resul_s, . . __W__K _A__aRIr_ __w_ _Ir_ _aw Ma___3 -- ADD THIS GEM TO YOUR RADAR AND WATCH TRADE ON MONDAY! -- DON'T EVEN BLINK! -- SWNM DOESN'T SLEEP! . . Removal_ _ Disclaimer_ _ -- Best regards Vibe

6 16

Spam in Images
by Alan Arndt Nov. 5, 2006

Nov. 5, 2006

Over the past month or more I have noticed a large increase in the amount of spam I receive with the Spam text translated into images. The actual text of the message is benign gibberish designed to pass Bayesian filters. They have even taken the step of inserting random bits into the image so that no two images have the same signature. I've received many multiple messages with the same fundamental image. I haven't thought of a decent way to filter these types of things. I hope someone else can and that it can get implemented into SpamBayes. Until recently I was very please that only a few messages a day (of 20-50) make it through. Does anyone have any good suggestions? I can certainly send plenty of samples if someone needs them Thanks, Alan Arndt alan five eighty two at do it now.com

10 18

image analyze
by Luigi Pugnetti Nov. 5, 2006

Nov. 5, 2006

Hello, I think there is an errore in the frame selection inside PIL_decode_parts(). for frame in ImageSequence.Iterator(image): # Assume the pixel with the largest value is the # background. bg = max(frame.histogram()) if bg < bgpix: image = frame bgpix = bg outside this loop image (in my opinion) will be always the last frame of original image. A part from this the algorithm could be better. From my test using stddev and sum2 yield better result. >From a (very) raw test you may use the attached code snippet. Of course all of these are very simple test and can be wrong bug you test failed for a pattern common few days ago (see the attached image) -- Luigi Pugnetti Symbolic S.p.A. V.le Mentana, 29 I-43100 Parma Italy Tel: +39 0521 708811 Fax: +39 0521 776190

2 1

Windows compatibility - OCR [was: Unwanted stock solicitations]
by Vibe Grevsen Nov. 5, 2006

Nov. 5, 2006

Hi friends, >> >> 1. Put ocrad 0.16 in the path > I have no experience with mingw but I compiled ocrad > using it and I'm using the result (without cygwin dll) with no problem, Ok, but note that the sources posted in spambayes-something was 0.15! New version 0.16 can be downloaded here: http://ftp.gnu.org/gnu/ocrad/ocrad-0.16.tar.bz2 According to the changelog the character recognition was improved. If you built a 0.16 exe without cygwin1.dll I would like to test it. Can you post it somewhere together with a short desciption of how it was built? "Pretty please with sugar on top". >> > Have you tried other ocr programs? >> >> No, not yet. >> >> Tony Meyer suggested Tesseract: > I built tesseract with no problem. ... > I tested few spam images and the results were poor. >> I mailed with NoSpam Today! Support (spamassasin based) before I chose SB. >> They were doing research on FuzzyOcr and ImageInfo. Maybe we could ask >> again about their results. I believe FuzzyOcr is gocr-based? > > Yes, they are using gocr. But as I said in my previous mail it has its > own problems. Ok then it has at least been tried ... >> Since the ocr is working with ocrad and - as you see below - I get very >> good results I will be moving on to the next area now. > You are lucky. My results are so so. Probably I get a reduction of a > 60/70% of spam with images (which in itself could be considered not bad) > but way too much spam is not stopped. I expect results to vary and it is too early in my testing to tell, but today SB caught 17 of 18 spams. I changed spam cutoff to 0.7 however that didn't even seem nescessary. My database contains 845 spams and 1411 hams. Zero false positives! >> I think it is far more beneficial to do more research into the actual processing >> as you commented elsewhere than to start the whole testing/tweaking all over >> again with a new ocr engine. Of course that is just my opinion... > > Yes and no. We need a decent ocr engine to start with than we may focus > on better image manipulation. Yes... >> At the moment spambayes have trouble with image for the following >> reason: > > - PIL sometimes fail to handle the image. I'm still investigating the > issue but the images seems reasonably correct (IE, Firefox and many > viewers, on linux and windows, are able to display them). It's quite > rare and not a big issue Not an ocr problem, but a preprocessing problem... It's great that you are looking into this because I for one don't know python well enough to debug such issues. > - ocr results are poor. The worst case are when you get a sequenze of > chars (char space char space ...) or a long word. both are ignored by > spambayes Tokenizer problem, configurable. Not related to the ocr engine. > - There are images which contain more than words and in this case we may > get no tokens. I have seen many animations with moving artefacts. Usually not a problem, but it may be in the future. Again some filtering - which is preprocessing - might be a brilliant idea. > In few cases if the colors used inside the image are changed you get a > different result. We should work on filtering and histogram analysis to determine the correct threshold level for the ocr. If we find a better way than what ocrad already does then we can pass it via the -T parameter. Advanced filtering can even detect repetitive patterns or noise in the background and remove that. Sure a professional ocr engine like OmniPage Pro does huge amounts of preprocessing like i.e. automatic rotation correction etc. already, but that does not yet seem nescessary for our purpose... > I have no knowledge of image processing but I tried few simple > operations (like scaling, sharpening, convert to gray, ...) but I got no > results. They were all quick tests and the result are in no way > conclusive. I did a course in image analysis. I don't know python / PIL, but I could probably do some tests in Matlab when my numbers start to deteriorate. If you have a way to batch-extract images from emails or from a dbx-file or if you send me a zip of 100+ problematic spam images then I would be happy to run some tests i.e. on best scale factor and scaling algorithm. I can batch-convert them so only worry about extraction. > from my understanding in Options.py you set the default values, > bayescustomize.ini contain the values chosen by the user an in > Imagestripper.py the programmer may embed it's values ignoring the user > choice (joking) Something like that I think :) Did you try to change this in ImageStripper.py and did it make any change to the output? >> With 2 you should get this quality image tokens: >> >> watch >> out >> here >> comes >> the >> big >> one! ... >> That is about a 90% recognition or so. > Yes, sometimes the results are good and sometimes are much worst. In few > cases a scaling factor of 3 it's better. Just now I'm doing a retraining > with ocrad_scale set to 3. we will see in the next days if the result > are better or worst Yes, my initial suggestion was scaling by 4, but Skip argued to use 2. He did tests with different scales already. Intuitively a larger scale should be better. I found however that it slowed down the analysis. Now I don't know what ocrad does, but resampling might be better than resizing. Happy coding :) Vibe PS: How come my posts always show up as new threads? Using OE. Don't want to subscribe.

2 1

Windows compatibility - OCR [was: Unwanted
by Vibe Grevsen Nov. 4, 2006

Nov. 4, 2006

Hi friends, >> OCR code's now been tweaked and tested to work in both WinXP and >> Win9x. >> This should work in unix as well. >> >> Here is a summary: >> >> 1. Put ocrad 0.16 in the path > > As a note, for Windows you need a copy of ocrad with skip patch that > opens pnm files in binary mode otherwise ocrad will fail on a lot of > files. Actually you're probably refering to my "patch"? (Ocrad/CygWin1.dll) http://mail.python.org/pipermail/spambayes/2006-October/019983.html If you have MinGW experience - which I don't - I think you can compile an exe-only which don't need the dll. But then I don't know if it is actually working because of the POSIX emulation or they did change the source. (I did not...) You're right Skip pointed it out in the ocrad forum, but the developer was reluctant to change this then so I don't actually know why 0.16 is working... Just know it is, which is fine for me. > Have you tried other ocr programs? No, not yet. Tony Meyer suggested Tesseract: http://mail.python.org/pipermail/spambayes-dev/2006-September/003750.html but there seemed to be build issues... I haven't tried.. I mailed with NoSpam Today! Support (spamassasin based) before I chose SB. They were doing research on FuzzyOcr and ImageInfo. Maybe we could ask again about their results. I believe FuzzyOcr is gocr-based? > I tried gocr and I think that its result are somewhat better but version > 0.41 + pgm patch almost hangs Ok, probably needs some tweaking then. Since the ocr is working with ocrad and - as you see below - I get very good results I will be moving on to the next area now. I think it is far more beneficial to do more research into the actual processing as you commented elsewhere than to start the whole testing/tweaking all over again with a new ocr engine. Of course that is just my opinion... >> 5. Finally I sugest you change the default scale from 1 to 2 like in >> this line >> >> scale = options["Tokenizer", "ocrad_scale"] or 2 > > changing this surely doesn't hurt but ocrad_scale it's already set to 2 > in Options.py Ok, I missed that. Don't know which one has prevalence. ImageStripper.py, Options.py or bayescustomize.ini. With 2 you should get this quality image tokens: watch out here comes the big one! srrl about blow your minds add srrl your radar mon nov ob companu name: stellar resource new (otc bb:srrl.ob) sumbol: srrl prlce: tl._ targe_: tio skip:r 10 ueru s_rong buu our last feature, posted cains ouer __o_ the span weekithose- are ridiculous cainsl cet srrl nowl will makinc stunninc skip:a 10 next weekl massiue campaicns are about startl watch srrl trade monday nou obl don't left out! That is about a 90% recognition or so. > probably should be removed (or set to 2 as you suggest) Then I suggest removal as you say. Better avoid redundancy ( clutter :) ) Happy coding :) Vibe

2 1

knelt cruise ship
by Dobbin Combs Nov. 4, 2006

Nov. 4, 2006

1 0

Problem with POP3 Proxy: can't load Spambayes
by rcharmoz＠axxessgroup.com Nov. 4, 2006

Nov. 4, 2006

I am using SpamBayes POP3 Proxy Version 1.0.4 (March 2005) (binary), with version 2.3.5 (#62, Feb 8 2005, 16:23:02) [MSC v.1200 32 bit (Intel)] of Python; my operating system is Windows 5.1.2600.2 (Service Pack 2). I have trained 0 ham and 1 spam. The problem I am having is that I cannot make it work with outlook. I go to the configuration section, and load in all the arameters, but it does absolutely nothing

2 1