Mailman 3 July 2003 - Python-Dev

RE: [Spambayes] Question (or possibly a bug report)
by Meyer, Tony 24 Jul '03

24 Jul '03

> > So it's 0.0 < 0.001 that fails. > And specifically because "0.001" is being treated as exactly > 0. Yes. Changing the print to: print 0.0, 0.001, 0.0 < 0.001, 0.0 == 0.001, 0.0 > 0.001 gives: 0.0 0.0 False True False In addition, a print of: print 0.0, 0.001, 0.0 < 0.001, 0.0 == 0.001, 0.1, 0.01, 0.0001 gives: 0.0 0.0 False True 0.0 0.0 0.0 > Which locale was in effect? This is starting to get odd <wink>. German_Germany.1252 =Tony Meyer

1 0

RE: [Spambayes] Question (or possibly a bug report)
by Tim Peters 24 Jul '03

24 Jul '03

[Mark Hammond] > Hrm - OK - I bit the bullet, and re-booted as German locale. If I > remove all calls to setlocale(), I can provoke come *very* strange > math errors. > Both: > > File "E:\src\spambayes\Outlook2000\manager.py", line 664, in score > return self.bayes.spamprob(bayes_tokenize(email), evidence) > File "E:\src\spambayes\spambayes\classifier.py", line 236, in > chi2_spamprob > S = ln(S) + Sexp * LN2 > exceptions.OverflowError: math range error Can you investigate this one a bit deeper? My guess is that S *= 1.0 - prob in the loop before is treating 1.0 as 10.0 due to the .pyc-file locale-dependent loading problem I detailed earlier, and that S is overflowing to infinity as a result. Printing S inside the loop would shed best light on this, and printing S when the OverflowError occurs would nail it: >>> import math >>> S = 1e200 * 1e200 >>> S 1.#INF >>> math.log(S) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: math range error >>> Not all platforms will raise OverflowError on log(infinity), but Windows Python does. No platform other than Windows displays infinity as 1.#INF, by the way (that's what MS C produces).

2 3

RE: [Spambayes] Question (or possibly a bug report)
by Meyer, Tony 24 Jul '03

24 Jul '03

> > File > "E:\src\spambayes\Outlook2000\dialogs\AsyncDialog.py", line 45, > > in set_stages > > assert (abs(start_pos-1.0)) < 0.001, \ > > AssertionError: Proportions must add to 1.0 (1.0,(('', 1.0),)) > > That one is still baffling to me, for reasons explained in > great detail before (if none of the float literals in this > file get screwed, *or* if all of them get screwed by locale, > this assert shouldn't fail -- locale has to be screwing up > some of the float literals but not others, and that > possibility remains baffling to me). Note that I checked in > a new version of this assert that displays more info when it fails. I tried with that version, and then added a print of my own. I get: 1.0 0.0 0.0 0.0 False Traceback (most recent call last): [...] File "D:\cvs\spambayes\spambayes\Outlook2000\dialogs\AsyncDialog.py", line 46, in set_stages assert abs(start_pos-1.0) < 0.001, ( AssertionError: Proportions must add to 1.0 (1.0,(('', 1.0),),0.0) I added as line 45: print start_pos, start_pos-1.0, abs(start_pos-1.0), 0.001, 0.0 < 0.001 So it's 0.0 < 0.001 that fails. Got to go to a meeting now, will try more afterwards. =Tony Meyer

2 1

RE: [Spambayes] Question (or possibly a bug report)
by Tim Peters 24 Jul '03

24 Jul '03

[Mark Hammond] > ... > Hrm - OK - I bit the bullet, and re-booted as German locale. If I > remove all calls to setlocale(), I can provoke come *very* strange > math errors. Both: > > ... > - and in the "training" dialog, I can reproduce the original bug: > > File "E:\src\spambayes\Outlook2000\dialogs\AsyncDialog.py", line > 45, in set_stages > assert (abs(start_pos-1.0)) < 0.001, \ > AssertionError: Proportions must add to 1.0 (1.0,(('', 1.0),)) That one is still baffling to me, for reasons explained in great detail before (if none of the float literals in this file get screwed, *or* if all of them get screwed by locale, this assert shouldn't fail -- locale has to be screwing up some of the float literals but not others, and that possibility remains baffling to me). Note that I checked in a new version of this assert that displays more info when it fails.

1 0

RE: [Spambayes] Question (or possibly a bug report)
by Tim Peters 24 Jul '03

24 Jul '03

[Mark Hammond] > ... > Googling for this, I found a mail Tim should find either interesting > or terrifying - it mentions not only locale issues, but floating point > exception masks: > http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=utf-8&threadm= > %23MuH%23LU8BHA.2344%40tkmsftngp04 Heh. I'm only surprised the guy got pointed to printer drivers so fast! In Dragon's core speech recognition engine, we had mounds of paranoid code assuming that any call to anything we didn't write would screw up the FPU control registers. Driver writers seem to believe they need to fiddle every bit the hardware exposes, but they're incompetent to do so <wink>. Python has ignored this issue so far -- just as it's ignored almost all 754 issues. I'm surprised that nobody in PythonLand has ever reported a bug of this nature. Maybe those who experience it just write it off to general Windows flakiness ... but it shouldn't matter to spambayes. The fp arithmetic in spambayes should never overflow, underflow, or divide by 0, no matter how extreme the inputs -- and I know what each of the bits in the FPU control registers actually do <wink>.

1 0

RE: [Spambayes] Question (or possibly a bug report)
by Tim Peters 24 Jul '03

24 Jul '03

[copying to Python-Dev because it relates to a recent thread there too] [Meyer, Tony] > No doubt I should ask this on python-list not here, but since Tim > would probably be the one to answer it anyway... :) Why does the > setlocale() function give that impression? If LC_NUMERIC should > always be "C", shouldn't setlocale(LC_NUMERIC, x) raise some sort of > exception? Reading the locale module docs should make it clearer. If it's still unclear after reading the docs, ask again <wink>. For a concrete example of why it's still useful to "pretend" to change LC_NUMERIC, see below (the locale module functions are sensitive to the change, and the code below couldn't be written otherwise). > Maybe Outlook is at fault here? I've certainly seen that some of the > Outlook/COM/MAPI calls make changes to the locale. In particular, > mapi.MAPILogonEx() does - it changes the locale to whatever Outlook > (i.e. Windows) thinks it is. Could this then be screwing things up > for us? Oh yes. A .pyc file contains a compiled form of Python code objects. Part of what's in a .pyc file is the "marshal" form of numeric literals referenced by the source code. It so happens that marshal stores float literals as strings (repr-style). The unmarshaling code (executed when a .pyc file is loaded) is written in C, and uses C's atof() to convert these strings back to C doubles. atof() is locale-sensitive, so can screw up royally if LC_NUMERIC isn't "C" at the time a module is loaded. Here's a little program we can use to predict this kind of damage: """ import marshal, locale def damage(afloat, lcnumeric): s = marshal.dumps(afloat) print repr(afloat), "->", repr(s) # Now emulate unmarshaling that under a given locale. # Strip the type code and byte count. assert s[0] == 'f' raw = s[2:] print "Under locale %r that loads as" % lcnumeric, locale.setlocale(locale.LC_NUMERIC, lcnumeric) print repr(locale.atof(raw)) locale.setlocale(locale.LC_NUMERIC, "C") """ For example, running damage(0.001, "German") displays: """ 0.001 -> 'f\x050.001' Under locale 'German' that loads as 1.0 """ while damage(0.001, "C") displays what Python needs to happen instead: """ 0.001 -> 'f\x050.001' Under locale 'C' that loads as 0.001 """ So all kinds of bad things *can* happen. I'm still baffled by the spambayes logfile, though, because the failing assert is here: def set_stages(self, stages): self.stages = [] start_pos = 0.0 for name, prop in stages: stage = name, start_pos, prop start_pos += prop self.stages.append(stage) assert (abs(start_pos-1.0)) < 0.001, \ "Proportions must add to 1.0 (%g,%r)" % (start_pos, stages) and the failing call is here: self.set_stages( (("", 1.0),) ) Under a locale that ignores periods, string -> double 0.0 -> 0.0 1.0 -> 10.0 0.001 -> 1.0 So the assert above would act like assert (abs(start_pos-10.0)) < 1.0, \ and the call would act like self.set_stages( (("", 10.0),) ) start_pos - 10.0 would still be 0 then, and the assert should not fail.

1 0

posixpath.ismount() on non-posix systems
by Jack Jansen 24 Jul '03

24 Jul '03

I've just filed bug #776207: test_posixpath fails on MacOS9. The problem is that it expects posixpath.ismount('/') to return true. But on OS9 it returns false. I'm unsure as to the correct behaviour of this on non-posix systems, does anyone have a suggestion? How does it behave on Windows? Should test_posixpath maybe only run some of the tests on non-posix systems? -- Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

6 7

improved zipimport
by Thomas Heller 23 Jul '03

23 Jul '03

Is it too late to get this small patch in? <http://www.python.org/sf/775637> I'm hesitating to raise the priority... Thomas

4 9

Cannot import...
by Apurv Anand 23 Jul '03

23 Jul '03

Hi, Need Help. I am writing a wrapper for a small .c function using SWIG The problem i am facing is that I am not able to import the module generated by SWIG. Is there any special setting to be done to import custom modules in Windows (Windows2000 to be specific) Below is the error that appears: File "<stdin>", line 1, in ? File "example.py", line 4, in ? import _example ImportError: No module named _example thanks, apurv

2 1

Until Monday...
by Guido van Rossum 23 Jul '03

23 Jul '03

I'm going into the intense phase of our move now; my internet will soon be disconnected. I'll resurface Monday in California. In order to avoid having a huge backlog of email, I'm turning off delivery on my python-dev and python-checkins subscriptions. If you send a message guido(a)python.org, I should be able to get it Monday afternoon. I wish everyone, but especially Barry and the remaining two at PythonLabs, well with the release; you have my cell phone number! --Guido van Rossum (home page: http://www.python.org/~guido/)

4 7