Mailman 3 April 2011 - Python-ideas

Re: [Python-ideas] [Python-Dev] the role of assert in the standard library ?
by Tarek Ziadé April 28, 2011

April 28, 2011

This is a thread from python-dev I am moving to python-ideas, because I want to debate the "assert" keyword :) On Thu, Apr 28, 2011 at 12:27 PM, Michael Foord <fuzzyman(a)voidspace.org.uk> wrote: > On 28/04/2011 09:34, Terry Reedy wrote: >> >> On 4/28/2011 3:54 AM, Tarek Ziadé wrote: >>> >>> Hello >>> >>> I removed some assert calls in distutils some time ago because the >>> package was not behaving correctly when people were using Python with >>> the --optimize flag. In other words, assert became a full part of the >>> code logic and removing them via -O was changing the behavior. >>> >>> In my opinion assert should be avoided completely anywhere else than >>> in the tests. If this is a wrong statement, please let me know why :) >> >> My understanding is that assert can be used in production code but only to >> catch logic errors by testing supposed invariants or postconditions. It >> should not be used to test usage errors, including preconditions. In other >> words, assert presence or absence should not affect behavior unless the code >> has a bug. But it does affect the behaviour at the end: when the code has a bug, then the way the code works is affected and differs depending if -O is used. Let me take an example: bucket = [] def add(stuff): bucket.append(stuff) def purge_and_do_something(): bucket.clear() assert len(bucket) == 0 ... do something by being sure the bucket is empty ... here, we could say assert is legitimate, as it just checks a post-condition. But this code is not thread-safe and if it's run via several threads, some code could add something in the bucket while purge() check for the assertion. So, if I run this code using -O it will not behave the same: it will seem to work. I am not arguing against the fact that this code should be changed and set up a lock. My point is that I do not understand why there are assert calls to check post-conditions. Those seem to be relics from the developer that marked something she needed to take care of in her code, but did not yet. e.g. like a TODO or a XXX or a NotImplementedError Moreover, why -O and -OO are removing assertions in that case ? even if the assert is "supposed not to happen" it's present in the code and can happen (or well, remove it). So "optimize" potentially changes the behaviour of the code. >From what I understood so far, a line with an assert should be considered as a dead code if we want the code to behave always the same way. I remove dead code in my code... Cheers Tarek -- Tarek Ziadé | http://ziade.org

4 3

Equality of same NaN instances?
by ZS April 28, 2011

April 28, 2011

Last debate on python-dev about behavior NaNs in containers raised another important question: Should the instance of NaN be equal to itself? P.S. Of course I suppose that different instances of NaN do not have to be equal. --- Zaur

4 3

Re: [Python-ideas] Proposal for new-style decorators
by Christophe Schlick April 27, 2011

April 27, 2011

On Tue, Apr 26, 2011 at 11:20 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > > There is no special syntax for defining decorators -- just normal nested > function or class definition syntax. To put it another way, Python does not > have decorator objects. A decorator is simply a callable (function, class, > or class instance with __call__ method) applied to a object I totally agree with that. When defining a decorating function, you don't have any syntactic element that could explain the reader of your code that this function is actually a decorator. It is only when applied on a function with the @-syntax that the mechanism becomes visible (but this is not always done in the same file). This can be considered as a strength (any function with a correct input/output is likely to later become a decorator, even if the original author did not thought about it). However, according to my own little experience, more than 9 times out of 10, you perfectly now when writting such a function that it is actually a decorator. Using the proposed @decorator idiom (hey see, I haven't written "syntax" ;-) has at least the advantage to be explicit when you want to be explicit (besides the other features it provides), > What you are actually proposing is a meta-decorator (class) whose instances > can be used as decorators because the class has a __call__ instance method. > This sort of thing is a known alternative to the nested function pattern. Yes, I know that using callable class can be an alternative to the nested function pattern. In the pattern you talk about, the __init__ method gets the decorator arguments, the __call__ method serves as a decorator making function, and a third method is used to generate the actual decorating function. As a result, the boilerplate is approximatively the same as with the nested functions idiom. But this is not what is done here, because the end-user only writes a single function, not a whole class. In my proposal, the two nested functions are avoided by the fact that (1) the decorator attributes are automatically injected as meta-attributes (this is the role of the middle nested function in the standard idiom), and (2) the decorating function is in charge to pass the whole set of arguments to the undecorated function (this is the role of the inner nested function in the standard idiom). As far as I know, I haven't seen the combination of these two elements before. > Stick with the two real problems. > > 1. The double or triple nested function pattern has a lot of boilerplate and > can be difficult to learn. Hiding boilerplate in a class makes the use > pattern simpler and easier to learn. This is a real benefit. > 2. Introspection (more comments below), which your class also addresses. OK. I'll drop the arguments on nested function, and simply focus on boilerplate and introspection. That makes sense. >> def old_style_repeat_var(n=3, trace=True): >> """docstring for decorating function""" > > Actually, this is the docstring for the decorator making function. Also agree. I've written "decorating function" by symmetry with the new idiom, but I knew that I would get some remark here ;-) >> '@wraps(func)' copies the name and the docstring from the undecorated >> function to the decorated one, in order to get some useful piece of >> information when using 'help'. However, the signature of the function >> still comes from the decorated function, not the genuine one. > > I am not sure what you mean. If the two signatures are different, then one > must use the signature of the wrapper when calling it, not the signature of > the wrappee, which is perhaps what you mean by 'the genuine > one'. The problem of generic wrappers having generic signatures (*args, > **kwds) is endemic to using generic wrappers instead of special case > wrappers. What I wanted to say is that wraps only does half of the job: it correctly copies the name and the docstring, but the signature presented by the help function is still test(*args, **keys), while it should actually be test(first=0, last=0) according to the undecorated function. The alternative proposed by the new idiom is to copy nothing at all: it simply says "OK, 'test' is a decorated function. If you want to know more look at 'test.func' to get the info about the undecorated one, and at 'test.deco' to see what the decorator has done". I prefer such a raw-but-explicit approach rather than an automatic half-baked, half-bogus one. Moreover, it is not easy with the 'functools' module to provide introspection of the decorating function, once you've got the decorated one. >> The only >> solution is to inspect the undecorated function and then use 'exec' to >> generate a wrapper with a correct signature. This is basically what is >> done in the 'decorator' package (available at PyPI) written by Michele >> Simionato. There has been a lengthy discussion in python-dev (in 2009 >> I guess, but I can't find the archive right now) whether to include or >> not this package in the standard library. > > The other solution is to not use a generic wrappers with generic signatures > but to write specific wrappers with the actual signature, which people did, > for instance, before functools and partial() were added to Python. Right again, but this overweights the boilerplate even more compared to the '@wraps' decorator, no? > I second the other recommendations to make your proposal > available on the cookbook site, etc. That sounds good... Thanks a lot, Terry!

3 6

Proposal for new-style decorators
by Christophe Schlick April 26, 2011

April 26, 2011

Hi Jim, The new version. Now cut-and-paste code should work... ------ Hello everybody, My name is Christophe Schlick, from the University of Bordeaux, France. I've been using Python for years, but it happens that this is my first post on a Python-related mailing list. For the first time, I feel that I may have some topic interesting enough for the Python community, so I would be very happy to get any kind of feeback (positive or negative) on it. Thanks in advance... The goal of this post is to propose a new syntax for defining decorators in Python. I would like to fully explain the rationale and the benefits of this proposal, as I guess that the discussion might be more fruitful if all details are written down once for all. However this has led me to a very long post (a kind of informal PEP) that was blocked by the moderators. So, following their recommendation, I've divided the text in three successive parts: (1) the rationale, (2) the proposal, (3) the proposed implementation and additional questions/remarks. For better clarity of the description, I will call the proposed syntax as "new-style decorators" (NSD, for short) and rename the classic syntax as "old-style decorators" (OSD), following the terms used some years ago with the (re)definition of classes. By the way, the introducing process for NSD shares many aspects with the process used for introducing new-style classes, including the following features: * No existing syntax is broken: the only thing required to create a new-style decorator is to decorate itself by a newly-introduced decorator called... "decorator" (well, this sentence is less recursive than it might appear at the first reading). * Every thing that can be done with OSD is possible with NSD, but NSD offer additional more user-friendly features. * NSD can peacefully live together with OSD in the same code. An NSD may even decorate an OSD (and vice-versa), however some properties of the NSD are lost with such a combination. -------------------------------------------------- 1 - Why bother with a new syntax? To explain what I don't like with the current syntax of decorators, let me take the example of a basic decorator (called 'old_style_repeat_fix') that simply repeats 3 times its undecorated function, and adds some tracing to the standard output. Here is the code: #--- def old_style_repeat_fix(func): """docstring for decorating function""" # @wraps(func) def dummy_func_name_never_used(*args, **keys): """docstring for decorated function""" print "apply 'old_style_repeat_fix' on %r" % func.__name__ for loop in range(3): func(*args, **keys) return dummy_func_name_never_used #--- Even if such code snippets have become quite usual since the introduction of decorators in Python 2.2, many people have argued (and I am obviously one of them) that the decorator syntax is a bit cumbersome. First, it imposes the use of nested functions, which often reduces readability by moving the function signature and docstring too far from the corresponding code. Second, as anonymous lambdas expressions can usually not be employed for decorating functions, the programmer has no other choice than to create a dummy function name (only used for one single 'return' statement), which is never a good coding principle, whatever the programming language. Once you have tried to teach decorators to a bunch of students, you really understand how much this syntax leverages the difficulty to grab the idea. The situation is even worse when the decorator needs some arguments: let's create an extended decorator (called 'old_style_repeat_var) that includes an integer 'n' to control the number of iterations, and a boolean 'trace' to control the tracing behavior. Here is the code: #--- def old_style_repeat_var(n=3, trace=True): """docstring for decorating function""" def dummy_deco_name_never_used(func): """docstring never used""" # @wraps(func) def dummy_func_name_never_used(*args, **keys): """docstring for decorated function""" if trace: print "apply 'old_style_repeat_var' on %r" % func.__name__ for loop in range(n): func(*args, **keys) return dummy_func_name_never_used return dummy_deco_name_never_used #--- This time a two-level function nesting is required and the code needs two dummy names for these two nested functions. Note that the docstring of the middle nested function is even totally invisible for introspection tools. So whether you like nested functions or not, there is some evidence here that the current syntax is somehow suboptimal. Another drawback of OSD is that they do not gently collaborate with introspection and documentation tools. For instance, let's apply our decorator on a silly 'test' function: #--- @old_style_repeat_var(n=5) # 'trace' keeps its default value def test(first=0, last=0): """docstring for undecorated function""" print "test: first=%s last=%s" % (first, last) #--- Now, if we try 'help' on it, we get the following answer: #--- >>> help(test) dummy_func_name_never_used(*args, **keys) docstring for decorated function #--- which means that neither the name, nor the docstring, nor the signature of the 'test' function are correct. Things are a little better when using the 'wraps' function from the standard 'functools' module (simply uncomment the line '@wraps(func)' in the code of 'old_style_repeat_var'): #--- >>> help(test) test(*args, **keys) """docstring for undecorated function""" #--- '@wraps(func)' copies the name and the docstring from the undecorated function to the decorated one, in order to get some useful piece of information when using 'help'. However, the signature of the function still comes from the decorated function, not the genuine one. The reason is that signature copying is not an easy process. The only solution is to inspect the undecorated function and then use 'exec' to generate a wrapper with a correct signature. This is basically what is done in the 'decorator' package (available at PyPI) written by Michele Simionato. There has been a lengthy discussion in python-dev (in 2009 I guess, but I can't find the archive right now) whether to include or not this package in the standard library. As far as I know, there is currently no clear consensus whether this is a good idea or not, because there has always been a mixed feeling from the community about transparent copy from the undecorated to the decorated function (even about the 'wraps' function): on one hand, transparent copy is cool for immediate help, for automatic documentation and for introspection tools, but on the other hand, it totally hides the decorating process which is not always what is wanted... or needed. The syntax for NSD presented in this proposal tries to improve this situation by offering two desirable features, according to the Zen of Python: * "flat is better than nested": no nested functions with dummy names are required, even when parameters are passed to the decorator; only one single decorating function has to be written by the programmer, whatever the kind of decorator. * "explicit is better than implicit": introspection of a decorated function explicitely reveals the decoration process, and allows one to get the name/signature/docstring not only for the corresponding undecorated function, but also for any number of chained decorators that have been applied on it. ------ to be continued in Part 2... CS

1 0

('blue', 'red', 'orange' if something, 'green')
by cool-RR April 25, 2011

April 25, 2011

Here's an idea that would have helped me today while coding. Allow something like this: ('blue', 'red', 'orange' if some_condition, 'green') So 'orange' is included in the tuple only if `some_condition` evaluates to `True`. This could be applied to list literals, tuple literals, set literals, and possibly dict literals, though the latter might be too clunky. I expect this to be rejected, but I really couldn't think of an elegant way to achieve the same thing with existing syntax. Ram.

14 21

random.boolean or bernoulli
by Erich Blume April 21, 2011

April 21, 2011

I am yet a novice python user, and not a strong statistician to boot, but I had an idea about how to enhance the 'random' module with one or two new functions for generating boolean values instead of floating-point values. This idea has a lot of flexibility in how it might be implemented, but I propose two new functions for the random module that might be implemented in python as follows def boolean(): return choice((True,False)) def bernoulli(p): return random() <= p It's true that since both of these functions have very simple short-statement implementations that it might be unnecessary baggage, but it occurred to me that their implementation might be consistent with the rest of the module. ~eblume

10 14

documentation: mention the version in which a syntax construct was added
by Mathias Panzenböck April 18, 2011

April 18, 2011

I wonder in which version the "except Foo as e" syntax was added (as opposed to "except Foo, e"). The documentations does not state this but I think it wasn't there in Python 2.4. I propose that such things are always mentioned in the documentation and formated so that you can easily find the version numbers. See how Mozilla marks such things (scroll down): https://developer.mozilla.org/en/DOM/range -panzi

5 5

Re: [Python-ideas] [Python-Dev] python and super
by Michael Foord April 17, 2011

April 17, 2011

On 16/04/2011 00:38, Greg Ewing wrote: > Michael Foord wrote: > >> consider the "recently" introduced problem caused by object.__init__ > > not taking arguments. This makes it impossible to use super correctly > > in various circumstances. > > > > ... > > >> It is impossible to inherit from both C and A and have all parent >> __init__ methods called correctly. Changing the semantics of super as >> described would fix this problem. > > I don't see how, because auto-super-calling would eventually > end up trying to call object.__init__ with arguments and fail. > No, not as I described. Where a method does not call up to its parent class then super would *not* auto call the parent class (I'm *not* suggesting a fully auto-call super as the original poster did), but a method not calling super still wouldn't halt the chain of calls. To achieve this a call into a method that doesn't call super (and would normally end the chain) instead removes itself and its unique parents [1] from the mro for this call. It would make the semantics more complex, but it would solve this problem and the original posters problem. For this specific example, if no classes call up to object.__init__ then it won't be called. It would permit you to have methods that can participate in multiple inheritance whilst still blocking calls (overriding) up to their parent classes. All the best, Michael [1] i.e. classes that aren't also the parent of another class still in the mro. > You might think to "fix" this by making a special case of > object.__init__ and refraining from calling it. But the same > problem arises in a more general way whenever some class in > the mix has a method with the right name but the wrong > signature, which is likely to happen if you try to mix > classes that weren't designed to be mixed together. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

3 5

Simple class initialization
by Adam Matan April 17, 2011

April 17, 2011

0. Abstract =========== A class initialization often begins with a long list of explicit variable declaration statements at the __init__() method. This repetitively copies arguments into local data attributes. This article suggests some semi-automatic techniques to shorten and clarify this code section. Comments and responses are highly appreciated. 1. Credit ========= The idea emerged from my question at stackoverflow.com. I would like to thank all those who answered and commented on that thread. http://stackoverflow.com/questions/1389180 2. The problem ============== Consider the following class: class Process: def __init__(self, pid, ppid, cmd, fd, reachable, user): If the instance needs to hold these arguments internally, local data attributes are declared and the value of the argument is being copied, using two mainstream notations: a. self.pid=pid self.ppid=ppid self.cmd=cmd self._fd=fd self.reachable=reachable self.user=user b. self.pid, self.ppid, self.cmd, self._fd, self.reachable, self.user = pid, ppid, cmd, fd, reachable, user a. takes an unreasonable amount of lines and has a repetative form. b. is long and prone to errors, especially when the argument list changes during development. 3. Solution outline =================== 1. Generally comply with the Zen of Python. 1. Explicit. The instance should not store any value unless told. 2. Short. 3. Backward compatible. Current Class syntax should work with the new solution. 4. Readable and intuitive. 5. Flexible. There should be a way to store any given subset of the arguments. 6. Allow storage of "private" variables by adding a single or double underscore before the variable name. 4. Solutions ============ 4.1 Decorator ------------- Nadia Alramli suggested this at the aforementiond thread at stackoverflow: from functools import wraps import inspect def initializer(fun): names, varargs, keywords, defaults = inspect.getargspec(fun) @wraps(fun) def wrapper(self, *args): for name, arg in zip(names[1:], args): setattr(self, name, arg) fun(self, *args) return wrapper class Process: @initializer def __init__(self, pid, ppid, cmd, fd, reachable, user) Pros: Simple, short, explicit and intuitive. Easy to add to the standard library, fully backward-compatible. Cons: Stores all arguments. Does not support private data attributes notation (underscore prefix). See http://stackoverflow.com/questions/1389180/python-automatically-initialize-… 4.2. Argument tagging --------------------- Arguments that needed to be stored within the instance could be marked with a special character, e.g. '~'. The character would be placed after the argument name for private variables: class Process: def __init__(self, ~pid, ~ppid, ~cmd, fd~, ~reachable, ~user) Pros: Simple, short and explicit. Can store any subset of the arguments. Supports private variable notation. Cons: Not intuitive. Changes the method signature and might be confusing. 4.3 Standard function --------------------- A function will be called to store the argument as data attributes. class Process: def __init__(self, pid, ppid, cmd, fd, reachable, user) acquire(pid, ppid, cmd, reachable, user) acquire(fd, prefix='_') Possible keywords can ba acquire, store, absorp. Pros: Explicit, clear and intuitive. Cons: Long - especially if more than a single prefix is used. 4.4 Initialization list ----------------------- The argument list would include the name of the local data attribute, a separator, and the argument name. class Process: def __init__(self, pid:pid, ppid:ppid, cmd:cmd, _fd:fd, reachable:reachable, user:user) """ pid, ppid, cmd, reachable and user are stored as data properties with the same name. fd is stored as _fd.""" Or: class Process: def __init__(self, :pid, :ppid, :cmd, _fd:fd, :reachable, :user) """Same, but local data attributes with the same name as arguments would be stored without stating their name twice.""" This is a developed argument tagging (4.2). Pros: See 4.2 Cons: Alters the method signature Not intuitive. Looking forward for comments, Adam matan

11 14

parameter annotation semantics
by Bruce Leban April 17, 2011

April 17, 2011

Subject was: Re: [Python-ideas] Simple class initialization On Sat, Apr 16, 2011 at 6:27 PM, Steven D'Aprano <steve(a)pearwood.info>wrote: > Bruce Leban wrote: > > One >> of the strange aspects (to me) of parameter annotations is that they have >> no >> semantics which opens them up to multiple conflicting uses. >> > > That's not a bug, that's a feature. > > It's been stated many times by Guido that it's far too early to standardize > on a single meaning for annotations. (We may *never* standardize on a single > meaning.) Instead, it is up to the library or decorator to impose whatever > meaning makes sense for that particular library or decorator. > I understand that. I can still think it a bit strange, can't I? To be more specific, you hit the crux of the problem with the statement "it is up to * THE* library or decorator to impose whatever meaning makes sense" [emphasis added] is that it assumes the singular. If I want to use two decorators which impose different meanings, I'm stuck. Imagine I have two decorators like this: @memoize # memoizes return values; if memo_value annotation is used, memoizes using # the value returned by applying the function def a(foo:memo_value(int), bar:memo_value(tuple)): pass @log_me # logs function calls and return values excluding parameters annotated no_log def b(foo, bar:no_log): pass Can I use both @memoize and @logging? Not if they're defined as they are above. I can if the decorators expect a list of annotations instead of a solo annotation: @memoize def a(foo:[memo_value(int)], bar:[memo_value(tuple)]): pass @log_me def b(foo, bar:[no_log]): pass @memoize @log_me def c(foo:[memo_value(int)], bar:[no_log, memo_value(tuple)]): pass Note that I am NOT proposing a change to the language (none is necessary), just suggesting that this would be good advice to offer authors. PEP 8 offers no advice on annotation syntax and PEP 3107 explicitly does not offer any advice saying "Despite yet more discussion, it was decided not to standardize a mechanism for annotation interoperability." What I would propose is something more like this: There is no standard for annotation interoperability. Annotations that wish to support interoperability should consider treating the annotation as an enumerable of values and checking value in annotation (or something similar) instead of value == annotation. Of course an obvious alternative is @memoize @log_me def c(foo:{memoize:memo_value(int)}, bar:{log_me:no_log, memoize:memo_value(tuple)}): pass but I think that's too verbose and therefore less likely to be adopted. --- Bruce *New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April Fools! *New!** *Blog post: http://www.vroospeak.com Ironically, a glaring Google grammatical error

6 5