Mailman 3 April 2004 - Python-Dev

RE: [Python-Dev] More imformative iterator representations
by Robert Brewer 07 Apr '04

07 Apr '04

Raymond Hettinger wrote: > Armin's note reminded me that the only thing I do not like about using > iterators is not being able to see their contents at the interactive > prompt. So, I posted an idea about changing the print part of the > read-eval-print loop to display more information: > > >>> enumerate('abcdefgh') > <enumerate object at 0x401a102c: (0, 'a') (1, 'b') (2, 'c') ...> > > There a couple of problems with the approach. One is how to > identify an > iterator object -- the presence of __iter__() and next() is a good but > not perfect indicator. Oh, I don't know. The Library Ref makes it pretty clear: "The iterator objects themselves are required to support the following two methods, which together form the iterator protocol: __iter__() ... next() " If not perfect, it's still a reasonable assumption. Since there's no Iterator superclass from which to inherit, you're going to have to rewrite each repr by hand anyway, and therefore can decide each one independently. Are there reasonable iterator objects which don't follow the "iterator protocol"? > An alternative to the read-eval-print approach is providing a custom > __repr__() method for the builtin iterators. +1. > Q: Are there any other candidate output formats? > A: Yes, we could add length information for a more mathematical style > output: > > reversed(['h', 'g', 'f', ... to 8 objects]) The very presence of __len__ might be a strong argument for providing the new repr, and its absence an indicator to avoid such. Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

4 3

RE: [Python-Dev] Re: PEP 318: Decorators last before colon
by Robert Brewer 07 Apr '04

07 Apr '04

[Guido van Rossum] > > ...while far from perfect, in > > terms of readability setting function attributes in a > decorator is so > > much better than setting them after the function has been defined, > > that I believe we have no choice but to provide it. [Me] > The more I read statements like this, the more I think of replacing: > > def foo(arg1, arg2): > pass > foo = classmethod(foo) > foo.attrA = 3 > > ...with code like: > > foo = classmethod() > foo.attrA = 3 > def foo(arg1, arg2): > pass > > Why would such additional features not be standard attributes of > function objects? >8 > All in all, I'd prefer that function decorators and attributes use > simple, known techniques like assignment and callables (as > shown above), > than become monolithic, crystallized all-in-one statements like: > > def foo(arg1 as str, arg2 as int) returns str [classmethod] > {attrA: 3}: > > I understand the desire to keep all those bits near each other on the > page, but stronger is my desire to make them independent > statements. If > we could bind a "blank" function object and supply its actual > codeblock > in a later statement, the rest falls magically into place, IMO. I sat down and banged out a metaclass solution, just to see if I *could* get the syntax I wanted in the example Shoe class, and to see what might be required behind the scenes. I put Shoe and all the decorator junk in one module to make the syntax look as clean as if Decorable, etc. were builtins. Basically, Decorable (the metaclass) looks at all attributes of the Shoe class; if there's an attribute "x" and an attribute "x_suite", then Shoe.x gets rebound as x(x_suite). The example provides classmethod and arbitrary function attributes. There are ways to clean it up, of course; this was a quick and dirty proof-of-concept. Rather than playing with "_suite", it should probably either reuse the same name "x" or use a token other than def. I also arbitrarily chose to allow a tuple or list for the decorator (but didn't demonstrate it). ----decorate.py---- import new class Decorable(type): def __init__(cls, name, bases, dct): type.__init__(cls, name, bases, dct) for attr in dct: if attr + "_suite" in dct: # Wrap the attr_suite in the attr decorator. inner_func = getattr(cls, attr + "_suite") decorator = getattr(cls, attr) if isinstance(decorator, (tuple, list)): wrapped_func = inner_func for item in decorator: wrapped_func = item.decorate(wrapped_func) else: wrapped_func = decorator.decorate(inner_func) setattr(cls, attr, wrapped_func) # [Optional] Unbind the inner function from the class. delattr(cls, attr + "_suite") class Decorator(object): def __init__(self): self.attributes = {} def __setattr__(self, key, value): if key == 'attributes': self.__dict__['attributes'] = value else: self.__dict__['attributes'][key] = value def decorate(self, function): for key, value in self.attributes.iteritems(): function.__setattr__(key, value) return function class clsmethod(Decorator): def decorate(self, function): newfunc = new.instancemethod(function, function.im_class(), function.im_class()) return super(clsmethod, self).decorate(newfunc) class Shoe(object): __metaclass__ = Decorable sized_shoe = clsmethod() sized_shoe.author = "Robert E Brewer" def sized_shoe_suite(cls, size): """Make a new Shoe with a 'size' attribute.""" newShoe = Shoe() newShoe.size = size return newShoe ----end decorate.py---- >>> import decorate >>> dir(decorate.Shoe) ['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__metaclass__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'sized_shoe'] >>> decorate.Shoe.sized_shoe.author 'Robert E Brewer' >>> s = decorate.Shoe.sized_shoe(3) >>> s <decorate.Shoe object at 0x0115CAD0> >>> s.size 3 Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

RE: [Python-Dev] More imformative iterator representations
by Robert Brewer 07 Apr '04

07 Apr '04

Bob Ippolito wrote: > > Raymond Hettinger wrote: > >> Q: Are there any other candidate output formats? > >> A: Yes, we could add length information for a more > mathematical style > >> output: > >> > >> reversed(['h', 'g', 'f', ... to 8 objects]) > > > > The very presence of __len__ might be a strong argument for > providing > > the new repr, and its absence an indicator to avoid such. > > This is essentially getting back to what Armin was saying.. if > iterators act more like lists, they will be easier to user and will > open up some new doors for optimization potential. Adding a > way to get > the length of some iterators essentially makes them list-like or > tuple-like. Agreed. I don't mind the crux of Armin's idea, I was reacting more to applying such optimizations to enumerate() in particular. Given sufficient clarification like the above, it'd be easier to swallow. Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

Some comments on PEP 328 (absolute/relative imports)
by Bernhard Herzog 07 Apr '04

07 Apr '04

I have some comments on PEP 328 (absolute/relative imports). A brief summary would be "-1", mostly because it unnecessarily breaks backwards compatibility and will cause extra work just to keep up with Python. Missing Rationale for the Incompatibility ----------------------------------------- First of all, I think the rationale for the absolute imports is missing some important points. >From the pep: In current Python, if you're reading a module located inside a package, it is not clear whether :: import foo refers to a top-level module or to another module inside the package. I don't think this is a good description of the actual problem. The problem is that the modules in the package shadow global ones and there's no easy way to bypass the local package. FWIW, though, I don't recall ever running into a problem with that. To resolve the ambiguity, it is proposed that ``foo`` will always be a module or package reachable from ``sys.path``. What I'm missing here is an explanation *why* this is the proposed solution. From my point of view the problem happens too rarely to justify breaking existing code. It's not even difficult to come up with a backwards compatible solution. In the few cases where a local module shadows a global one, the import could flaged as absolute with the global keyword, for instance (as has been suggested in one of the threads referenced in the pep): import global time from global foo import bar Unlike the proposed solution, this not only retains compatibility, it would also be similar to how conflicts between local and global scopes are handled elsewhere in Python. For instance, local variables hide globals and builtins. If you later need to access a shadowed global you need to e.g. rename your local variables or resort to some other way to get at the global. Why, then should local, i.e. relative, imports not shadow global ones by default unless a globel, i.e. absolute, import is explicitly requested? Python-Version Independent Code ------------------------------- I like relative imports and it's good to see that the pep proposes to keep them and even to extend their capabilities. However, I also like to write code that runs on a range of Python versions, e.g. 2.2 and later. Unfortunately, if, as is hinted at in the pep, the new syntax becomes the default in 2.5, there will no way to write relative imports that will work in both 2.3 and 2.5. So, for me the pep effectively abolishes relative imports completely. It also means that I will have to go through all my code and change relative imports to absolute imports, costing me at least a few hours that I would rather spend enhancing my code instead of just keeping up with Python. Deprecation Warnings -------------------- Another thing missing is a note about DeprecationWarnings. Since the change will break exisiting code, DeprecationWarnings should be given for relative imports at some point before the new semantics become the default. Leading Dots ------------ Leading dots were also recently mentioned to be used for a with-statement. Won't two different uses for leading dots be confusing for newbies? E.g. in with some_object: .a = 1 from .magic import xyzzy the leading dot in the import statement won't have anything to do with the with statement. I guess code like the above would be rare, but some people expressed concern about the use of "as" for decorators for similar reasons. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/

4 7

PEP 318: How I would implement decorators
by Jim Hugunin 07 Apr '04

07 Apr '04

I've spent a couple days reading the decorators discussion and thinking about the issues. That's not a lot of time, but I still thought I'd share my opinions on implementing this feature. This all is in my (mostly) humble opinion. When I started looking at PEP 318, I thought that the most important issue was deciding on the correct syntax. After a while I became much more worried about semantics. * Semantics, composition and understandable failures One of the nice properties of C# and Java attributes is that they are declarative rather than transformative so there are no conflicts between different attributes placed on the same object. This is probably too strong of a restriction for Python; nevertheless, it's a valuable reference point. Using the definition of returns from the PEP, I get the following behavior if I confuse its order with classmethod, i.e. class C: [classmethod(m), returns(str)] def m(cls): return repr(cls) C.m() --> TypeError: unbound method new_f() must be called with C instance as first argument (got nothing instead) This is a hard message to understand, but at least it does occur at the first instance of calling the method. Even worse is where you forget to create an instance of a class or to call a function like returns, i.e. class C: [returns] def m(self): return "hello" c = C() print c.m() #there's no error for this call --> function new_f at 0x01278BB0> This kind of error scares me a lot. It can be extremely hard to track down because it doesn't occur either at the point of declaration or the point of calling the method, but only at some later stage where the result from the method is used. I'm also concerned that this error is caused by the person using the attribute not the person writing it and that person may be less sophisticated and more easily confused. A less critical, but important issue is the propagation of doc strings and other function attributes, i.e. class C: [returns(str)] def m(self): "a friendly greeting" return "hello" print C.m.__doc__ --> None This will really mess up programs that rely on doc strings or other function attributes. After thinking about these issues, my first inclination was to sign up for Raymond Hettinger's NO on 318 campaign. Unfortunately, as my previous message showed there are a number of ways that I'd really like to use decorators in my Python code. My next thought was to propose that Python's decorators be restricted to be declarative like C# and Java attributes. They could be used to add meta information to functions, but other meta-tools would be needed for any actual transformations. I'm still open to that idea, but I've come to believe that's not a very pythonic approach. My current idea is that we should design this feature and any documentation around it to strongly encourage people to write safe, composable decorators. Specifically, it should be easier to write safe and composable decorators than to write bad ones. After that, we can rely on decorator developers to do the right thing. 1. Use a "decorate" method instead of a call to avoid the worst problems like [returns]. This has the added benefit in my eyes of making it a little harder to write small decorators and thus encouraging people to think a little more before writing these meta-programs. 2. Provide a small number of helper classes that make it easy to write safe and composable decorators. Here are my top two candidates: class attribute: def __init__(self, **kwds): self.attrs = kwds def decorate(self, decl): if not hasattr(decl, __attrs__): decl.__attrs__ = [] decl.__attrs__.append(self) return decl def __repr__(self): ... class func_wrapper: """subclass this and implement a wrap function""" def decorate(self, func): assert iscallable(func) new_func = self.wrap(func) assert iscallable(new_func) self.copy_attributes(func, new_func) return new_func def copy_attributes(self, old_func, new_func): ... func_wrapper would be used for things like synchronized and memoize, and attribute would be used for most of the other examples in my previous message. 3. Provide a clear set of guidelines for good decorators, i.e. a. decorators should not have side effects outside of the class/function b. decorators should return the same kind of thing they are given c. decorators should be commutative whenever possible (requires b) + sys._getframe() There should be a consensus on whether or not decorators like 'generic' are a good or a bad idea. If they're a good idea then the decorate method should get an extra argument exposing the local namespace so that sys._getframe() won't be required. I'd be in favor of adding that argument to decorate. + classmethod and staticmethod These two decorators are a big problem because they don't return the same kind of thing as they are given. This means they always have to be the last decorator applied in any list. That kind of restriction is never a good thing. Personally, I'd like to see these changed from returning a new object to instead setting an attribute on the existing function/method object. Then the method object could use this attribute to implement its __get__ method correctly. This would make these decorators compose very nicely with all of the other decorators; however, it would be a fairly major change to how Python handles static and class methods and I'm doubtful that would fly. * Syntax I've come to see syntax as less crucial than I originally thought. In particular, the examples I've read on this list of using temporary variables showed me that even if the syntax is too confining there are tolerable solutions. Nevertheless, I'm still strongly in favor of putting the decorator list in front of the def/class. Decorators/attributes are an exciting new technique that's gaining increasing use in different programming languages. They're important enough to be one of the key ideas that Java-1.5 is stealing from C# (turn-about is always fair play). The AOP community often has vehement debates not about the value of attributes, but about how they can be most effectively used. Because decorators are still a relatively new idea, it's hard to know how much room they will need. Certainly if we're only interested in classmethod and staticmethod it's not much. However, I think it's important to give decorators as much room as possible to be prepared for the future. I have some sympathy with the arguments that a bare prefix list is already legal Python syntax and is certainly found in existing code. I'm quite sure that using a bare list would break existing Python programs that have dangling lists left over as they evolved. I like Sam's proposal to add a '*' in front of the list. I could also be persuaded to go along with any of the new grouping constructing, like <| |>; however, those all seem like too much of a change for the value. I'm fairly confident that the *[] form would allow for attributes to optionally be put on the same line as the def. This would be LL1, but perhaps the Python parser has other restrictions that I'm unaware of, i.e. *[java_sig("public void meth(int a, int b)"), staticmethod] def meth(self, a, b): pass or for simple cases *[staticmethod] def meth(self, a, b): pass -Jim

3 2

Re: Rich comparisons
by Edward Loper 07 Apr '04

07 Apr '04

>>Tim Delaney's suggestion [1] seemed pretty reasonable. In particular: >> >> - Ensure that NaN is a singleton (like True and False). I.e., the >> float constructor checks if the float is NaN, and if so returns a >> singleton. > > And we do that... how, exactly? > > [...] > > Without wanting to sound patronizing or rude -- but risking it anyway > -- you're not giving the impression of understanding these issues any > more than I do, and I *know* I don't understand them well enough to > get them straight. You're right -- I apparently don't understand the issues well enough. And (worse) I didn't understand how far my understanding went. I was basing my proposal on Tim Delaney's suggestion and on a couple quick experiments on my 2 machines. Given that things aren't as simple as I had hoped they were, I guess I'll leave it up to people who know more to figure it out. But I'm not particularly happy about the fact that you have to be an expert to understand the intricacies of basic arithmetic with floats (not that I have a way to fix it). -Edward

15 48

Expert floats
by Jewett, Jim J 07 Apr '04

07 Apr '04

Ka-Ping Yee: > >>> 3.3 > 3.2999999999999998 > confused and frightened them, and continues to confuse and frighten > almost everyone i teach. It bothered me when I first saw it (not in Python). It was still better than trying to figure out why my code was doing the wrong thing, when my math and logic were both clearly correct. To steal Simon Percivall's example, I assumed 2.20 - 1.20 == 1.00. It didn't even occur to me to check something as obviously true as that -- until I had learned and remembered that floats were not decimals. -jJ

4 4

PEP 318: More examples of decorator use
by Jim Hugunin 06 Apr '04

06 Apr '04

I'm a little scared of getting involved in the discussion of PEP 318. Nevertheless, I think any language design discussion can benefit from more examples. Here's a list of ways that I would use decorators in my Python code if I had simpler syntax than is available today. *Interoperability with other languages and systems jythonc parses method docstrings looking for a special directive to generate Java-compatible method signatures, i.e. def meth(self, a, b): "@sig public void meth(int a, int b)" pass If Python had structured metadata, this would change to [java_sig("public void meth(int a, int b)")] def meth(self, a, b): pass C# uses this mechanism to provide interoperability with C, COM, and web services. Here's a translation of a web services example from the .NET framework docs. [WebService(Description="Common Server Variables", Namespace="http://www.contoso.com/")] class ServerVariables: [WebMethod(Description="Obtains the Server Computer Name", EnableSession=false)] def GetMachineName(self): return self.Server.MachineName [SoapDocumentMethod(Action="http://www.contoso.com/Sample", RequestNamespace="http://www.contoso.com/Request", RequestElementName="GetUserNameRequest", ResponseNamespace="http://www.contoso.com/Response", ResponseElementName="GetUserNameResponse"), WebMethod(Description="Obtains the User Name")] def GetUserName(self): ... This is the same functionality motivating Bob Ipolito's pleas for this feature to support pyobjc, and it would obviously also help IronPython. *Tracing and other global policies Aspect-oriented programming has had great success capturing global policies for things like tracing in a single module. For a language like Python that already has strong meta-programming support it's possible to program in an AOP style without any language changes (see the tracing implementation in parrotbench/b0.py for an example). However, one lesson learned from actually applying AOP to large systems (http://hugunin.net/papers/pra03-colyer.pdf) was that there was always a need to capture a few local exceptions. Current AOP systems can do this in a clumsy way, but almost everyone agrees that C#/Java attributes are the best way to specify these kinds of exceptions. Here's an example of a tracing policy with a local exception. from tracing import notrace, add_tracing class C: def m1(self): ... def m2(self): ... [notrace] def m0(self): ... if __DEBUG__: add_tracing(globals()) This code will recursively walk all the declarations in a module and wrap all methods and functions with tracing code. Attributes are used here to indicate special handling for specific methods. In this case, m0 should not be traced. *Unit testing The nunit test framework uses attributes instead of naming conventions to label test methods. My favorite part of this is the additional attributes, for example 'expected_exception' to indicate that the correct behavior for a test is to throw a particular exception, i.e. [test, expected_exception(InsufficientFundsError)] def transfer2(self): self.source.transfer(100000000000) vs. def test_transfer2(self): self.assertRaises(InsufficientFundsError, self.source.transfer, 1000000000) This example is extremely compelling for C# or Java code where functions aren't first class and a complicated try/catch block is required instead of assertRaises. It's less compelling than I'd expected for Python; however, I still find the version with attributes a little more clear. *Parsing There are many simple parser implementations for Python that embed grammar rules in doc strings. This would be nicer to have in explicit metadata. Here's an example from Brian Sabbey's dparser for python. [action("exp", "exp '+' exp")] def d_add(t): return t[0] + t[2] [action("exp", '"[0-9]+"')] def d_number(t): return int(t[0]) *Synchronization I think a synchronized decorator would be useful, but more sophisticated locking policies are even more interesting. Here's an example of reader/writer synchronization assuming a metaclass ReadWriteSynchro that cooperates with attributes on specific methods. class DataStore(ReadWriteSynchro): [reader] def read(self, key): .... [writer] def write(self, key, value): ... ... *Other examples that I'd probably use memoized from Michael Chermside to automatically cache the results of idempotent functions accepts and returns type checking wrappers from the PEP until/unless Python gets a better answer for true optional type declarations generic from Edward Loper to support generic functions. I both really like and really hate this decorator. I'd love to have tolerable syntax for generic functions and this gives me something tolerable. On the other hand, I really dislike code that uses sys._getframe() so that might be enough to scare me off of this. *Some examples that I'd like to see in an anti-patterns list onexit from the PEP. I don't think decorators should have side-effects visible outside of the class/function itself. This seems MUCH better accomplished with the explicit call to the registration function. singleton from the PEP. Whenever possible, decorators should return the same kind of thing that they are given. This code seems much less clear than explicitly creating a singleton instance. propget, propset, propdel from Edward Loper via Guido. To me, these violate the principle of modularity. I much prefer the explicit property declaration that puts all of the parts together. class C(object): def getX(self): return self.__x def setX(self, x): self.__x = x x = property(getX, setX) -Jim

2 1

Python Interest Group
by Bill Venners 06 Apr '04

06 Apr '04

Hello All, I am the editor of Artima.com. Guido and others suggested that I send this query to python-dev. I am about to release a new feature on Artima called Interest Groups, and I am looking for a few people who would like to moderate a Python Interest Group. I am wondering if any of you might like to serve as a moderator, or could recommend others from the Python community who might want to act as moderators. Here is the low down on Interest Groups: Artima Interest Groups is a free service that allows communities to keep informed and discuss issues. Each interest group is essentially a news feed that serves a specific community. The news feed is moderated by one or more volunteers from the community. Anyone can submit news items for consideration. The moderators check the accuracy of the submissions, and decide which news items to approve. Approved items appear on web pages at Artima.com, and in two Artima RSS feeds. Each approved item can be discussed by the community in its own Artima Forums topic. Basically, the moderators would get a short email notification when someone submits an item, then they can edit, approve, or reject the item via the web. It is like having a little theserverside.com or slashdot.org for your community. Another way you might think of it is as a web version of python-announce, except that only the topics are moderated. Any discussion that follows each topic is a free-for-all. One benefit of having an Interest Group at Artima is that people outside the Python community proper will also be exposed to the Python community's news. If anyone is interested, or has a recommendation of someone I can approach about moderating a Python Interest Group, please email me. Thanks. Bill -- Bill Venners Artima Software, Inc. http://www.artima.com

1 0

Python is faster than C
by Jewett, Jim J 06 Apr '04

06 Apr '04

Dave Cole: > Would it make sense to build specialisation into the base Python > interpreter so that it builds specialised versions of bytecode for > functions. I am assuming that there are only a few types that can be > targeted by specialisation (int, float). Bytecodes then could be added > to the interpreter which operate on those native machine types rather > than the Python objects. If you add too many specializations, you blow the cache. If you add not enough, then you help only certain types of programming. If you're willing to do that, you might consider going even farther, as in http://www.foretec.com/python/workshops/1998-11/proceedings/papers/aycock-21 1/aycock211.html Basically, he creates a large number of pseudo-codes that just mean "do this, then this, then this" without going back to the eval loop in between. Because he knows what is coming next, there are additional chances for optimizations. -jJ

1 0