Mailman 3 April 2003 - Python-Dev

Python Developers
by Neeta Ramchandani 24 Apr '03

24 Apr '03

Hi, I know there aren't many of you guys, but I have an Investment Bank that is looking for an OO Scriptor, with at least 2 years Java experience with Unix and proper Python development skills. Anyone know anyone....or anyone interested in this 3-6 contract? Neeta Ramchandani Key Account Manager Harvey Nash IT Investment Banking / Finance Team DD: 020 73331518 Fax: 020 73332657 E-mail: nramchandani(a)harveynash.com Website: www.harveynash.com ***************************************************************** IMPORTANT NOTICE The information in this e-mail and any attached files is CONFIDENTIAL and may be legally privileged or prohibited from disclosure and unauthorised use. The views of the author may not necessarily reflect those of the Company. It is intended solely for the addressee, or the employee or agent responsible for delivering such materials to the addressee. If you have received this message in error please return it to the sender then delete the email and destroy any copies of it. If you are not the intended recipient, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication or any action taken or omitted to be taken in reliance upon this message or its attachments is prohibited and may be unlawful. At present the integrity of e-mail across the Internet cannot be guaranteed and messages sent via this medium are potentially at risk. All liability is excluded to the extent permitted by law for any claims arising as a result of the use of this medium to transmit information by or to the Harvey Nash Group plc. *****************************************************************

2 1

heapq
by Agthorr 24 Apr '03

24 Apr '03

Hello, I'm new to this list, so I will begin by introducing myself. I'm a graduate student at the University of Oregon working towards my PhD. My primary area of research is in peer-to-peer networks. I use Python for a variety of purposes such as constructing web pages, rapid prototyping, and building test frameworks. I have been a Python user for at least two years. I must confess that I have not lurked on the list much before making this post. I did search back in the list though, so in theory I won't be bringing up a rehashed topic... Recently, I had need of a heap in Python. I didn't see one in the 2.2 distribution, so I went and implemented one. Afterwards, I wondered if this might be useful to others, so I decided to investigate if any work had been done to add a heap to Python's standard library. Low and behold, in CVS there is a module called "heapq". I compared my implementation with heapq, and I see some important differences. I'm not going to unilaterally state that mine is better, but I thought it would be worthwhile to raise the differences in this forum, so that an informed decision is made about The Best Way To Do Things. Hopefully, it will not be too terribly controversial :) The algorithms used are more or less identical, I'm primarily concerned with the differences in interface. As written, heapq provides some functions to maintain the heap priority on a Python list. By contrast, I implemented the heap as an opaque class that maintains a list internally. By creating this layer of abstraction, it is possible to completely change the heap implementation later, without worrying about affecting user programs. For example, it would be possible to switch to using Fibonacci Heaps or the Pairing Heaps mentioned by Tim Peters in this message: http://mail.python.org/pipermail/python-dev/2002-August/027531.html Another key difference is that my implementation supports the decrease_key() operation that is important for algorithms such as Dijkstra's. This requires a little extra bookkeeping, but it's just a small constant factor ;) For the API, my insert() function returns an opaque key that can later be used as a parameter to the adjust_key() function. For those who like looking at source code, my implementation is here: http://www.cs.uoregon.edu/~agthorr/heap.py -- Dan Stutzbach

8 20

Updating PEP 246 for type/class unification, 2.2+, etc.
by Phillip J. Eby 24 Apr '03

24 Apr '03

I'd like to propose some revisions to PEP 246 based on experience trying to implement a prototype of it for use in PEAK and Zope (perhaps Twisted as well). The issues I see are as follows: 1. PEP 246 allows TypeError in __conform__ and __adapt__ methods to pass silently. (After considerable work and thought, I was able to reverse-engineer *why*, but that rationale should at least be explicitly documented in the PEP, even if the limitation is unavoidable.) 2. The reference implementation in the PEP has fancy extra features that are not specified by the main body of the PEP, and in some cases raise more questions than they answer about what a valid PEP 246 implementation should do. (adaptRaiseTypeException, adaptForceFailException, _check, etc.) 3. The PEP 246 examples do not illustrate Python 2.2+ idioms for creating usable __conform__ and __adapt__ methods. For example, a class instance with a __call__ method gets stuck in another class in order to (presumably) work around the absence of staticmethod or classmethod in Python prior to version 2.2. The reference implementation also uses string exceptions, which were a no-no even before version 2.2. 4. PEP 246 does not cover implementation issues for developers in the cases where 'obj' is a class or 'protocol' is an instance. The former is particularly important in the context of adapting metaclass instances, and the latter is relevant for using Zope 'Interface' objects (for example) as protocols. None of these issues are unresolvable; in fact I have proposals to address them all. If the PEP authors agree with my assessments, perhaps they will undertake to update the PEP. My goal is not to get a PEP 246 'adapt()' blessed for the Python core or distro in the immediate future, but rather to have a usable reference standard for framework developers to build implementations on. Even more important... I would like framework users to be able to write __conform__ and __adapt__ methods that will be in principle usable by any framework that uses PEP 246 as a standard for adaptation. In this sense, we may view the role of PEP 246 as being similar to the Python DBAPI. So, without further ado, my proposals for revisions to PEP 246 are as follows: Issue #1: My reverse-engineering leads me to the conclusion that PEP 246 specifies that TypeError be ignored because of issue #4 above: using a class as 'obj' or an instance for 'protocol' may lead to a TypeError caused by using a class method as an instance method or vice versa. While the creator of the objects being supplied to 'adapt()' can work around these issues with descriptors, the casual user should not be expected to. Thus, such TypeErrors should be ignored. To resolve this dilemna, I propose that 'adapt()' use the following pseudocode to verify whether a TypeError has arisen from invocation of a method, or the execution of a method: try: # note: real implementation needs to catch AttributeError! result = obj.__conform__(protocol) if result is not None: return result except TypeError: if sys.exc_info()[2].tb_frame is not sys._getframe(): raise In other words, if the exception was raised in the calling frame, it is assumed to be an invocation error rather than an execution error, and can thus be safely ignored. The only "exception" to this pattern is if the targeted method is written in C and thus does not create a separate frame for execution. (Note that C code generated by Pyrex creates dummy execution frames before returning an exception to Python, so this is only an issue for hand-written C code.) The worst case scenario here is that authors of '__conform__' and '__adapt__' methods written in C must 1) guarantee that TypeError will not be raised, 2) accept silent loss of internal TypeErrors, or 3) write code to create a dummy frame when raising an error. As far as Jython impact, the mechanism by which TypeErrors are raised is different, so I do not know if it is possible for the Java or Python levels to cleanly make this differentiation. If Jython simulates Python frames and tracebacks, including only Python-level frames, then this would work more or less directly. I confess I do not understand enough about Jython's implementation at present to know how practical it is under Jython. An alternative might be to recognize the text of the Python exception values for unbound methods, missing arguments, etc., applying to the method being called. This might actually be more complex to implement correctly, though. Issue #2: I propose that the PEP 246 reference implementation be pared down to remove extraneous features. Specifically, I believe that the signature of adapt should be: _marker = object() def adapt(obj, protocol, default=_marker): # ... attempt to return adapted result if default is _marker: raise NotImplementedError(...) 'adaptForceFailException' looks to me like a YAGNI, since an object shouldn't veto its being used for a protocol, if the protocol knows how to adapt it. And the protocol doesn't need to force failure, it can return failure. Rather than raising a TypeError for adaptation failure, and thus "raising" even further confusion regarding the proper handling of TypeError. Finally, the '_check()' function should be dropped. Its presence simply makes it harder to evaluate or consider PEP 246 for inclusion in Python or a framework, because it is left unspecified what '_check()' should do. We are given many examples of what it *could* do, but not what it *should* do. In any event, I think it's a YAGNI because if the object claims it can conform or the protocol claims it can adapt, then what business is it of 'adapt()' to question the consent of the objects involved? Issue #3: Examples should use 'classmethod' for '__adapt__' rather than simulated or real 'staticmethod', and include the case where a subclass delegates to a superclass '__adapt__' method. And string exceptions are right out. Issue #4: Illustrate the issues that arise for adapting classes or metaclass instances, and using instances rather than types as protocols. Ideally, examples of descriptors that work around the issues should be included. (And as soon as I've figured out how to write them, I'll be happy to supply source!) Thoughts, anyone?

1 0

okay to beef up tests on the maintenance branch?
by Skip Montanaro 23 Apr '03

23 Apr '03

Is it okay to beef up the test harness code a little on the 2.2 maintenance branch? I'm installing 2.2.2 on a Solaris 8 machine at the moment and notice two small warts: * -u all isn't accepted * there are no sunos5 expected skips Any problem adding them for 2.2.3 if they aren't already in CVS (they may already be there)? More generally, is improving the test harness okay (not strictly a bug fix) since it doesn't directly affect the performance of the interpreter? Thx, Skip

2 1

Re: [Python-Dev] Metatype conflict among bases?
by Phillip J. Eby 23 Apr '03

23 Apr '03

David Abrahams <dave(a)boost-consulting.com> wrote: > >Consider: > > class A(object): > class __metaclass__(type): > pass > > class B(A): # TypeError: metatype conflict among bases > class __metaclass__(type): > pass > >Now that's a weird error message at least! There's only one base (A), >and I'm telling Python explicitly to use the nested __metaclass__ >instead of A's __metaclass__! > >Should I not be surprised that Python won't let me set the metatype >explicitly? The problem here is that B.__metaclass__ *must* be the same as, or a subclass of, A.__metaclass__, or vice versa. It doesn't matter whether the metaclass is specified implicitly or explicitly, this constraint must be met. Your code doesn't meet this constraint. Here's a revised example that does: class A(object): class __metaclass__(type): pass class B(A): class __metaclass__(A.__class__): pass B.__metaclass__ will now meet the "metaclass inheritance" constraint. See the "descrintro" document for some more info about this, and the "Putting Metaclasses To Work" book for even more info about it than you would ever want to know. :) Here's a short statement of the constraint, though: A class X's metaclass (X.__class__) must be identical to, or a subclass of, the metaclass of *every* class in X.__bases__. That is: for b in X.__bases__: assert X.__class__ is b.__class__ or issubclass(X.__class, b.__class__),\ "metatype conflict among bases"

4 4

Fwd: summing a bunch of numbers (or "whatevers")
by Alex Martelli 23 Apr '03

23 Apr '03

Sorry to distract python-dev's august collective attention from its usual exhalted concerns down to a mundane issue;-), but... we may be able to strike a tiny blow for simplicity, clarity, power, AND performance at once... For the Nth time, today somebody asked in c.l.py about how best to sum a list of numbers. As usual, many suggested reduce(lambda x,y:x+y, L), others reduce(int.__add__,L), others reduce(operator.add,L), etc, and some (me included) a simple total = 0 for x in L: total = total + x The usual performance measurements were unchained (easier than ever thanks to timeit.py of course;-), and the paladins of reduce were once again dismayed by the fact that the best reduce can do (that best is obtained with operator.add) is mediocre (e.g. on my box with L=range(999), reduce takes 330 usec, and the simple for loop takes 247). Discussion proceeded on whether "reduce(operator.add, L)" was abstruse for most people, or not, and on whether the loop was or wasn't "too low level", as the Pythonic approach to such a common task. It then struck me that Python doesn't HAVE "one single obvious way" to do what IS after all a rather common task in everyday programming, namely, "sum up this bunch of things" (typically numbers, occasionally strings -- and when they're strings the "obvious" loop above is terribly slow, a typical newbie trap...). Somebody proposed having operator.add take any number of arguments -- not quite satisfactory, AND dog-slow, it turned out to be (when I tried a quick experimental mod to operator.c), due to the need to turn a sequence (typically a list) into a tuple with *. Now, I think the obvious approach would be to have a function sum, callable with any non-empty homogeneous sequence (sequence of items such that + can apply between them), returning the sequence's summation -- now THAT might help for simplicity, clarity AND power. So, I tried coding that up -- just 40 lines of C... it runs twice as fast as the plain loop, for summing up range(999), and just as fast as ''.join for summing up map(str, range(999)) [for the simple reason that I special-case this -- when the first item is a PyBaseString_Type, I delegate to ''.join]. Discussing this with newbie-to-moderately experienced Pythonistas, the uniform reaction was on the order of "you mean Python doesn't HAVE a sum function already?!" -- most everybody seemed to feel that such a function WOULD be "the obvious way to do it" and that it should definitely be there. So -- by this time I'm biased, having invested a bit of time in this -- what do y'all think... any interest in this? Should I submit it? I'm not quite sure where it should go -- a builtin seems most natural (to keep company with min and max, for example), but maybe that would be too ambitious, and it should be in math or operator instead... Alex

26 69

RE: [Python-Dev] os.path.walk() lacks 'depth first' option
by Noah Spurrier 23 Apr '03

23 Apr '03

I like your version; although I used a different name to avoid confusion with os.path.walk. Note that os.listdir does not include the special entries '.' and '..' even if they are present in the directory, so there is no need to remove them. Tim Peters> for path in walk(root): Tim Peters>Or it could look like Tim Peters> for top, names in walk(root): Tim Peters>or Tim Peters> for top, dirnames, nondirnames in walk(root): I like the idea of yielding (top, dirs, nondirs), but often I want to perform the same operations on both dirs and nondirs so separating them doesn't help that case. This seems to be a situation where there is no typical case, so my preference is for the simpler interface. It also eliminates the need to build two new lists from the list you get from os.listdir()... In fact, I prefer your first suggestion (for path in walk(root)), but that would require building a new list by prepending the basepath to each element of children because os.listdir does not return full path. So finally in this example, I just went with returning the basepath and the children (both files and directories). Following Tom Good's example I added an option to ignore symbolic links. It would be better to detect cycles or at least prevent going higher up in the tree. Tim Peters>obvious topdown argument, note a subtlety: when topdown is True, the caller Tim Peters>can prune the search by mutating the dirs list yielded to it. For example, This example still allows you to prune the search in Breadth first mode by removing elements from the children list. That is cool. for top, children in walk('C:/code/python', depthfirst=False): print top, children if 'CVS' in children: children.remove('CVS') Yours, Noah from __future__ import generators # needed for Python 2.2 # Inspired by Doug Fort from an ActiveState Python recipe. # His version didn't use recursion and didn't do depth first. import os def walktree (basepath=".", depthfirst=True, ignorelinks=True): """This walks a directory tree, starting from the basepath directory. This is somewhat like os.path.walk, but using generators instead of a visit function. One important difference is that walktree() defaults to DEPTH first with optional BREADTH first, whereas the os.path.walk function allows only BREADTH first. Depth first was made the default because it is safer if you are going to be modifying the directory names you visit. This avoids the problem of renaming a directory before visiting the children of that directory. The ignorelinks option determines whether to follow symbolic links. Some symbolic links can lead to recursive traversal cycles. A better way would be to detect and prune cycles. """ children = os.listdir(basepath) if not depthfirst: yield basepath, children for name in children: fullpath = os.path.join(basepath, name) if os.path.isdir (fullpath) and not (ignorelinks and os.path.islink(fullpath)): for (next_basepath, next_children) in walktree (fullpath, depthfirst, ignorelinks): yield next_basepath, next_children if depthfirst: yield basepath, children def test(): for (basepath, children) in walktree(): for name in children: print os.path.join(basepath, name) if __name__ == '__main__': test()

3 3

Re: Magic number needs upgrade
by Guido van Rossum 23 Apr '03

23 Apr '03

> Now that we have new bytecode optimizations, the pyc file magic > number needs to be changed. Of course we might also consider turning back Raymond's bytecode optimizations. Given that I can't discern any speedup, I wonder what the wisdom is of adding more code complexity. We're still holding off on Ping and Aahz's changes (see the cache-attr-branch) and Thomas and Brett's CALL_ATTR optimizations, for similar reasons (inconclusive evidence of speedups in real programs). What makes Raymond's changes different? I also wonder why this is done unconditionally, rather than only with -O. --Guido van Rossum (home page: http://www.python.org/~guido/)

2 2

Metatype conflict among bases?
by David Abrahams 23 Apr '03

23 Apr '03

Consider: class A(object): class __metaclass__(type): pass class B(A): # TypeError: metatype conflict among bases class __metaclass__(type): pass Now that's a weird error message at least! There's only one base (A), and I'm telling Python explicitly to use the nested __metaclass__ instead of A's __metaclass__! Should I not be surprised that Python won't let me set the metatype explicitly? -- Dave Abrahams Boost Consulting www.boost-consulting.com

2 1

New re failures on Windows
by Tim Peters 22 Apr '03

22 Apr '03

Sorry, I can't make time for this. test_re is failing today: """ C:\Code\python\PCbuild>python ../lib/test/test_re.py Running tests on re.search and re.match Running tests on re.sub Running tests on symbolic references Running tests on re.subn Running tests on re.split Running tests on re.findall Running tests on re.match Running tests on re.escape Pickling a RegexObject instance Test engine limitations maximum recursion limit exceeded Running re_tests test suite === grouping error ('^((a)c)?(ab)$', 'ab', 0, 'g1+"-"+g2+"-"+g3', 'None-None-ab' ) 'None-a-ab' should be 'None-None-ab' """ test_sre is dying with a segfault: """ C:\Code\python\PCbuild>python ../lib/test/test_sre.py Running tests on character literals Running tests on sre.search and sre.match sre.match(r'(a)?a','a').lastindex FAILED expected None got result 1 sre.match(r'(a)(b)?b','ab').lastindex FAILED expected 1 got result 2 sre.match(r'(?P<a>a)(?P<b>b)?b','ab').lastgroup FAILED expected 'a' got result 'b' Running tests on sre.sub Running tests on symbolic references Running tests on sre.subn Running tests on sre.split Running tests on sre.findall Running tests on sre.finditer Running tests on sre.match Running tests on sre.escape Running tests on sre.Scanner Pickling a SRE_Pattern instance Test engine limitations """ and it dies with a segfault there. Unfortunately, test_sre doesn't die in a debug build.

5 10