Mailman 3 February 2005 - Python-Dev

Confusing "hasattr" behaviour
by J. David Ibanez 24 Feb '05

24 Feb '05

Python 2.4 (#1, Feb 22 2005, 20:15:07) [GCC 3.3.5 (Gentoo Linux 3.3.5-r1, ssp-3.3.2-3, pie-8.7.7.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class A(object): ... def get_x(self): ... there_is_a_bug_here ... x = property(get_x, None, None, '') ... >>> a = A() >>> getattr(a, 'x') Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 3, in get_x NameError: global name 'there_is_a_bug_here' is not defined >>> hasattr(a, 'x') False Today I have spent a while to hunt down a couple of bugs in my application, because "hasattr" was catching the exceptions. I see there was a patch (#504714, three years ago) that addressed this, but it was rejected because much code would get broken. But, is it going to be considered for sometime in the future? 3.0 maybe? And, would it really break so much code? Wouldn't it instead bring to the day light many bugs that are already there (but failing in a more silent way)? Thanks, -- J. David Ibáñez Itaapy <http://www.itaapy.com> Tel +33 (0)1 42 23 67 45 9 rue Darwin, 75018 Paris Fax +33 (0)1 53 28 27 88

3 2

RE: [Python-Dev] Re: PEP 754
by Warnes, Gregory R 24 Feb '05

24 Feb '05

[After a long delay, the thread continues....] Hi All, I'm pushing ahead on the tasks necessary to add the 'fcponst' module described in PEP 754: IEEE 754 Floating Point Special Values. Per http://www.python.org/psf/contrib, I've - Changed the license to the Apache License, Version 2.0 - Just faxed the "Contributor Agreement" to the PSF I've also - created a patch on sourceforge.net for fpconst code and documentation (see https://sourceforge.net/tracker/index.php?func=detail&aid=1151323&group_id=5 470&atid=305470) I will need help connecting the included test functions into the python test suite. What else needs to be done to allow fpconst to go into the Python library? -Greg > -----Original Message----- > From: Tim Peters [mailto:tim.one@comcast.net] > Sent: Friday, December 10, 2004 1:43 PM > To: Warnes, Gregory R; goodger(a)python.org > Cc: peps(a)python.org > Subject: RE: [Python-Dev] Re: PEP 754 > > > [Warnes, Gregory R] > > Hi David & Tim, > > > > First, I really like to see this go forward. The fpconst module is > > getting alot of use across the net, and it would be very useful to > > finally get it into the standard python library. What > needs to be done > > to move forward? > > Looks to me like exactly the same stuff as was needed before. > Guido needs > to pronounce on it. It needs a patch on SourceForge, adding > the new module, > new docs, and a test suite run by Python's standard regrtest.py. The > non-PSF copyright and license remain problematic. That last should be > easier to deal with after the PSF approves a Contributor > Agreement (probably > within a month), but will be a show-stopper if Pfizer won't > contribute the > module under the terms of the Contributor Agreement (which > can't be answered > now by anyone, since the Contributor Agreement doesn't exist yet). > > > LEGAL NOTICE Unless expressly stated otherwise, this message is confidential and may be privileged. It is intended for the addressee(s) only. Access to this E-mail by anyone else is unauthorized. If you are not an addressee, any disclosure or copying of the contents of this E-mail or any action taken (or not taken) in reliance on it is unauthorized and may be unlawful. If you are not an addressee, please inform the sender immediately.

1 0

Some old patches
by Reinhold Birkenfeld 23 Feb '05

23 Feb '05

Hello, this time working up some of the patches with beards: - #751943 Adds the display of the line number to cgitb stack traces even when the source code is not available to cgitb. This makes sense in the case that the source is lying around somewhere else. However, the original patch generates a link to "file://?" on the occasion that the source file name is not known. I have created a new patch (#1144549) that fixes this, and also renames all local variables "file" in cgitb to avoid builtin shadowing. - #749830 Allows the mmap call on UNIX to be supplied a length argument of 0 to mmap the whole file (which is already implemented on Windows). However, the patch doesn't apply on current CVS, so I made a new patch (#1144555) that does. Recommend apply, unless this may cause problems on some Unices which I don't know about. - #547176 Allows the rlcompleter to complete on [] item access (constructs like sim[0].<TAB> could then be completed). As comments in the patch point out, this easily leads to execution of arbitrary code via __getitem__, which is IMHO a too big side effect of completing (though IPython does this). Recommend reject. - #645894 Allows the use of resource.getrusage time values for profile.py, which results in better timing resolution on FreeBSD. However, this may lead to worse timing resolution on other OS, so perhaps the patch should be changed to be restricted to this particular platform. - #697613 -- bug #670311 This handles the problem that python -i exits on SystemExit exceptions by introducting two new API functions. While it works for me, I am not sure whether this is too much overhead for fixing a glitch no one else complained about. - #802188 This adds a specific error message for invalid tokens after a '\' used as line continuation. While it may be helpful when the invalid token is whitespace, Python usually shows the exact location of the invalid token, so you can examine this line and find the error. On the other hand, the patch is no big deal, so if a specific error message is welcome, it may as well be applied. Enough for today... and best of all: I have no patch which I want to promote! Reinhold

3 4

What about CALL_ATTR?
by Reinhold Birkenfeld 23 Feb '05

23 Feb '05

While rummaging in the old patches, I found this: """ The result of the PyCore sprint of me and Brett: the CALL_ATTR opcode (LOAD_ATTR and CALL_FUNCTION combined) that skips the PyMethod creation and destruction for classic classes (but not newstyle classes, yet.) The code is somewhat rough yet, it needs commenting, some renaming, and most importantly testing. It seems to work, however, and provides between a 35% and 5% speedup. (5% in 'average' code, up to 35% in instance method calls and instance creation alone.) It also needs to be updated to include newstyle classes. I will likely work on this on the flight home. """ (patch #709744) How is the status of this? Sounds promising, I'd say... Reinhold -- Mail address is perfectly valid!

2 1

RE: [Python-Dev] Windows Low Fragmentation Heap yields speedup of ~15%
by Gfeller Martin 23 Feb '05

23 Feb '05

> A well-known trick is applicable in that case, if Martin thinks it's > worth the bother: > grow the list to its final size once, at the start (overestimating if > you don't know for sure). Then instead of appending, keep an index to > the next free slot, same as you'd do in C. Then the list guts never > move, so if that doesn't yield the same kind of speedup without using > LFH, list copying wasn't actually the culprit to begin with. I actually did that in Py2.1, and removed it when we switched to Zope 2.7/Py 2.3, because it was no longer worth it, presumably due to obmalloc becoming enabled. Unfortunately, I have lost the speedup gained by my Fast-Append-List in Py 2.1, but I recall it saved about 5% in a similar test than the one I have today. > Yes. For example, a 300-character string could do it (that's not > small to obmalloc, but is to LFH). Strings produced by pickling are > very often that large, and especially in Zope (which uses pickles > extensively under the covers -- reading and writing persistent objects > in Zope all involve pickle strings). With my amateur (in C-stuff) knowledge, this sounds as a very likely point. >From Evan's 17 Feb mail, it sounds that cPickle would not use obmalloc - if this is the case, wouldn't that be an obvious (and relatively easy) speedup? > If someone were motivated enough, it would probably be easiest to > run Martin's app on a Linux box, and use the free Linux tools to analyze it. As it is often the case, real-life figures do not come from a benchmark, but from an application test with real data, so putting the whole thing up on Linux would need quite some time - which I don't have :-( Best regards, Martin -----Original Message----- From: Tim Peters [mailto:tim.peters@gmail.com] Sent: Friday, 18 Feb 2005 23:52 To: Evan Jones Cc: Gfeller Martin; Martin v. Löwis; Python Dev Subject: Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15% [Tim Peters] ... >> Then you allocate a small object, marked 's': >> >> bbbbbbbbbbbbbbbsfffffffffffffffffffffffffffffff [Evan Jones] > Isn't the whole point of obmalloc No, because it doesn't matter what follows that introduction: obmalloc has several points, including exploiting the GIL, heuristics aiming at reusing memory while it's still high in the memory heirarchy, almost never touching a piece of memory until it's actually needed, and so on. > is that we don't want to allocate "s" on the heap, since it is small? That's one of obmalloc's goals, yes. But "small" is a relative adjective, not absolute. Because we're primarily talking about LFH here, the natural meaning for "small" in _this_ thread is < 16KB, which is much larger than "small" means to obmalloc. The memory-map example applies just well to LFH as to obmalloc, by changing which meaning for "small" you have in mind. > I guess "s" could be an object that might potentially grow. For example, list guts in Python are never handled by obmalloc, although the small fixed-size list _header_ object is always handled by obmalloc. >> One thing to take from that is that LFH can't be helping list-growing >> in a direct way either, if LFH (as seems likely) also needs to copy >> objects that grow in order to keep its internal memory segregated by >> size. The indirect benefit is still available, though: LFH may be >> helping simply by keeping smaller objects out of the general heap's >> hair. > So then wouldn't this mean that there would have to be some sort of > small object being allocated via the system malloc that is causing the > poor behaviour? Yes. For example, a 300-character string could do it (that's not small to obmalloc, but is to LFH). Strings produced by pickling are very often that large, and especially in Zope (which uses pickles extensively under the covers -- reading and writing persistent objects in Zope all involve pickle strings). > As you mention, I wouldn't think it would be list objects, since resizing > lists using LFH should be *worse*. Until they get to LFH's boundary for "small", and we have only the vaguest idea what Martin's app does here -- we know it grows lists containing 50K elements in the end, and ... well, that's all I really know about it <wink>. A well-known trick is applicable in that case, if Martin thinks it's worth the bother: grow the list to its final size once, at the start (overestimating if you don't know for sure). Then instead of appending, keep an index to the next free slot, same as you'd do in C. Then the list guts never move, so if that doesn't yield the same kind of speedup without using LFH, list copying wasn't actually the culprit to begin with. > That would actually be something that is worth verifying, however. Not worth the time to me -- Windows is closed-source, and I'm too old to enjoy staring at binary disassemblies any more. Besides, list guts can't stay in LFH after the list exceeds 4K elements. If list-copying costs are significant here, they're far more likely to be due to copying lists over 4K elements than under -- copying a list takes O(len(list)) time. So the realloc() strategy used by LFH _probably_ isn't of _primary)_ interest here. > It could be that the Windows LFH is extra clever? Sure -- that I doubt it moves Heaven & Earth to cater to reallocs is just educated guessing. I wrote my first production heap manager at Cray Research, around 1979 <wink>. > ... > Well, it would also be useful to find out what code is calling the > system malloc. This would make it easy to examine the code and see if > it should be calling obmalloc or the system malloc. Any good ideas for > easily obtaining this information? I imagine that some profilers must > be able to produce a complete call graph? Windows supports extensive facilities for analyzing heap usage, even from an external process that attaches to the process you want to analyze. Ditto for profiling. But it's not easy, and I don't know of any free tools that are of real help. If someone were motivated enough, it would probably be easiest to run Martin's app on a Linux box, and use the free Linux tools to analyze it.

1 0

RE: [Python-Dev] Those import related syntax errors again...
by Samuele Pedroni 23 Feb '05

23 Feb '05

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

18 78

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 22 Feb '05

22 Feb '05

Patch / Bug Summary ___________________ Patches : 308 open (+10) / 2755 closed ( +1) / 3063 total (+11) Bugs : 838 open (+15) / 4834 closed ( +5) / 5672 total (+20) RFE : 168 open ( +0) / 148 closed ( +4) / 316 total ( +4) New / Reopened Patches ______________________ do not add directory of sys.argv[0] into sys.path (2004-05-02) http://python.org/sf/946373 reopened by wrobell isapi.samples.advanced.py fix (2005-02-17) http://python.org/sf/1126187 opened by Philippe Kirsanov more __contains__ tests (2005-02-17) http://python.org/sf/1141428 opened by Jim Jewett Fix to allow urllib2 digest auth to talk to livejournal.com (2005-02-18) http://python.org/sf/1143695 opened by Benno Rice Add IEEE Float support to wave.py (2005-02-19) http://python.org/sf/1144504 opened by Ben Schwartz cgitb: make more usable for 'binary-only' sw (new patch) (2005-02-19) http://python.org/sf/1144549 opened by Reinhold Birkenfeld allow UNIX mmap size to default to current file size (new) (2005-02-19) http://python.org/sf/1144555 opened by Reinhold Birkenfeld Make OpenerDirector instances pickle-able (2005-02-20) http://python.org/sf/1144636 opened by John J Lee webbrowser.Netscape.open bug fix (2005-02-20) http://python.org/sf/1144816 opened by Pernici Mario Replace store/load pair with a single new opcode (2005-02-20) http://python.org/sf/1144842 opened by Raymond Hettinger Remove some invariant conditions and assert in ceval (2005-02-20) http://python.org/sf/1145039 opened by Neal Norwitz Patches Closed ______________ date.strptime and time.strptime as well (2005-02-04) http://python.org/sf/1116362 closed by josh-sf New / Reopened Bugs ___________________ attempting to use urllib2 on some URLs fails starting on 2.4 (2005-02-16) http://python.org/sf/1123695 opened by Stephan Sokolow descrintro describes __new__ and __init__ behavior wrong (2005-02-15) http://python.org/sf/1123716 opened by Steven Bethard gensuitemodule.processfile fails (2005-02-16) http://python.org/sf/1123727 opened by Jurjen N.E. Bos PyDateTime_FromDateAndTime documented as PyDate_FromDateAndT (2005-02-16) CLOSED http://python.org/sf/1124278 opened by smilechaser Function's __name__ no longer accessible in restricted mode (2005-02-16) CLOSED http://python.org/sf/1124295 opened by Tres Seaver Python24.dll crashes, EXAMPLE ATTACHED (2005-02-12) CLOSED http://python.org/sf/1121201 reopened by complex IDLE line wrapping (2005-02-16) CLOSED http://python.org/sf/1124503 opened by Chris Rebert test_os fails on 2.4 (2005-02-17) CLOSED http://python.org/sf/1124513 reopened by doerwalter test_os fails on 2.4 (2005-02-16) CLOSED http://python.org/sf/1124513 opened by Brett Cannon test_subprocess is far too slow (2005-02-17) http://python.org/sf/1124637 opened by Michael Hudson Math mode not well handled in \documentclass{howto} (2005-02-17) http://python.org/sf/1124692 opened by Daniele Varrazzo GetStdHandle in interactive GUI (2005-02-17) http://python.org/sf/1124861 opened by davids subprocess.py Errors with IDLE (2005-02-17) http://python.org/sf/1126208 opened by Kurt B. Kaiser subprocesss module retains older license header (2005-02-17) http://python.org/sf/1138653 opened by Tres Seaver Python syntax is not so XML friendly! (2005-02-18) CLOSED http://python.org/sf/1143855 opened by Colbert Philippe inspect.getsource() breakage in 2.4 (2005-02-18) http://python.org/sf/1143895 opened by Armin Rigo future warning in commets (2005-02-18) http://python.org/sf/1144057 opened by Grzegorz Makarewicz reload() is broken for C extension objects (2005-02-19) http://python.org/sf/1144263 opened by Matthew G. Knepley htmllib quote parse error within a <script> (2005-02-19) http://python.org/sf/1144533 opened by Allan Hoeltje No os.statvfs on FreeBSD (2005-02-21) http://python.org/sf/1145231 opened by Volker Stolz shutil.copystat() may fail... (2005-02-21) http://python.org/sf/1145257 opened by Petr Prikryl Strange behaviour concerning variable names (2005-02-22) CLOSED http://python.org/sf/1145950 opened by Felix Nawothnig bsddb3 build problems on FreeBSD (2.4 + 2.5) (2005-02-23) http://python.org/sf/1146231 opened by Andrew I MacIntyre Windows deadlock with PyEval_ReleaseLock (2005-02-22) http://python.org/sf/1147646 opened by Lou Montulli fix bsddb documentation psize -> pgsize (2005-02-22) http://python.org/sf/1149413 opened by Martin Mokrejs bssdb wrapper does not export some low-level functions (2005-02-23) http://python.org/sf/1149447 opened by Martin Mokrejs Bugs Closed ___________ Static library incompatible with nptl (2005-02-10) http://python.org/sf/1119860 closed by loewis PyDateTime_FromDateAndTime documented as PyDate_FromDateAndT (2005-02-16) http://python.org/sf/1124278 closed by bcannon Function's __name__ no longer accessible in restricted mode (2005-02-16) http://python.org/sf/1124295 closed by mwh Python24.dll crashes, EXAMPLE ATTACHED (2005-02-11) http://python.org/sf/1121201 closed by bcannon test_os fails on 2.4 (2005-02-17) http://python.org/sf/1124513 closed by doerwalter test_os fails on 2.4 (2005-02-17) http://python.org/sf/1124513 closed by loewis Strange behaviour concerning variable names (2005-02-22) http://python.org/sf/1145950 closed by mwh RFE Closed __________ Option to force variables to be declared (2005-02-14) http://python.org/sf/1122279 closed by rhettinger Line Numbers (2005-02-14) http://python.org/sf/1122532 closed by kbk Python syntax is not so XML friendly! (2005-02-18) http://python.org/sf/1143855 closed by bcannon IDLE line wrapping (2005-02-16) http://python.org/sf/1124503 closed by kbk

1 0

[Python-Dev] Re: [Python-checkins] python/dist/src/Python compile.c, 2.344, 2.345
by Tim Peters 22 Feb '05

22 Feb '05

[rhettinger(a)users.sourceforge.net] > Modified Files: > compile.c > Log Message: > Teach the peepholer to fold unary operations on constants. > > Afterwards, -0.5 loads in a single step and no longer requires a runtime > UNARY_NEGATIVE operation. Aargh. The compiler already folded in a leading minus for ints, and exempting floats from this was deliberate. Stick this in a file: import math print math.atan2(-0.0, -0.0) If you run that directly, a decent 754-conforming libm will display an approximation to -pi (-3.14...; this is the required result in C99 if its optional 754 support is implemented, and even MSVC has done this all along). But if you import the same module from a .pyc or .pyo, now on the HEAD it prints 0.0 instead. In 2.4 it still prints -pi. I often say that all behavior in the presence of infinities, NaNs, and signed zeroes is undefined in CPython, and that's strictly true (just _try_ to find reassuring words about any of those cases in the Python docs <wink>). But it's still the case that we (meaning mostly me) strive to preserve sensible 754 semantics when it's reasonably possible to do so. Not even gonzo-optimizing Fortran compilers will convert -0.0 to 0.0 anymore, precisely because it's not semantically neutral. In this case, it's marshal that drops the sign bit of a float 0 on the floor, so surprises result if and only if you run from a precompiled Python module now. I don't think you need to revert the whole patch, but -0.0 must be left alone (or marshal taught to preserve the sign of a float 0.0 -- but then you have the problem of _detecting_ the sign of a float 0.0, and nothing in standard C89 can do so). Even in 754-land, it's OK to fold in the sign for non-zero float literals (-x is always unexceptional in 754 unless x is a signaling NaN, and there are no signaling NaN literals; and the sign bit of any finite float except zero is already preserved by marshal).

1 0

Re: [Python-Dev] UserString
by Michel Pelletier 22 Feb '05

22 Feb '05

On Tuesday 22 February 2005 03:01 am, Guido wrote: > > BTW, there's *still* no sign from a PEP 246 rewrite. Maybe someone > could offer Clark a hand? (Last time I inquired he was recovering from > a week of illness.) Last summer Alex, Clark, Phillip and I swapped a few emails about reviving the 245/246 drive and submitting a plan for a PSF grant. I was pushing the effort and then had to lamely drop out due to a new job. This is good grant material for someone which leads to my question, when will the next cycle of PSF grants happen? I'm not volunteering and I won't have the bandwidth to participate, but if there are other starving souls out there willing to do the heavy lifting to help Alex it could get done quickly within the PSFs own framework for advancing the language. -Michel

1 0

textwrap wordsep_re
by Karl Chen 22 Feb '05

22 Feb '05

Hi, textwrap.fill() is awesome. Except when the string to wrap contains dates -- which I would like not to be broken. In general I think wordsep_re can be smarter about what it decides are hyphenated words. For example, this code: print textwrap.fill('aaaaaaaaaa 2005-02-21', 18) produces: aaaaaaaaaa 2005- 02-21 A slightly tweaked wordsep_re: textwrap.TextWrapper.wordsep_re = \ re.compile(r'(\s+|' # any whitespace r'[^\s\w]*\w+[a-zA-Z]-(?=[a-zA-Z]\w+)|' # hyphenated words r'(?<=[\w\!\"\'\&\.\,\?])-{2,}(?=\w))') # em-dash print textwrap.fill('aaaaaaaaaa 2005-02-21', 18) behaves better: aaaaaaaaaa 2005-02-21 What do you think about changing the default wordsep_re? -- Karl 2005-02-21 17:39

2 1