proto-PEP: Fixing Non-constant Default Arguments
The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert Title: Fixing Non-constant Default Arguments Abstract This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly. Motivation Currently, to write functions using non-constant default arguments, one must use the idiom: def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following: def foo(mutable=some_expr_producing_mutable): #rest of function However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate? Rationale Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were: def foo(bar=<baz>): #code def foo(bar=new baz): #code def foo(bar=fresh baz): #code def foo(bar=separate baz): #code def foo(bar=another baz): #code def foo(bar=unique baz): #code where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below. Specification The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter. Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a >>> foo() hi >>> CONST="bye" >>> foo() bye #...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)] >>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3) #default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b >>> bar() bbbb #default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c >>> baz() 2 >>> baz() 3 #...but only when they're required def silly(): print "spam" return 42 def qux(d=silly()): pass >>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls #Rule 3 count = 0 def next(): global count count += 1 return count - 1 def frobnicate(g=next(), h=next(), i=next()): print g, h, i >>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition Backwards Compatibility This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from: def foo(bar=mutable): #code to def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap @stateify(mutable) def foo(bar): #code or state = mutable def foo(bar=state): #code or class Baz(object): def __init__(self): self.state = mutable def foo(self, bar=self.state): #code The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used. In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs. Reference Implementation All code of the form: def foo(bar=some_expr, baz=other_expr): #body Should act as if it had read (in pseudo-Python): def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed. References [0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html [1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6 [2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 [3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/ [4] Function Annotations http://www.python.org/dev/peps/pep-3107/
Hello! I'd liked to say outright that this bad idea which complicates matters more than provides solutions. Right now it is enough to know that the part from def to ":" is executed at definition time. This is what incremental dynamic semantics is about. So, the suggestion is good only as separated feature, but is IMHO wrong if considered in the language design as a whole. So things like def foo(non_const=None): non_const = non_const or [] are good becuase explicitely tell you that the mutable object is to be created at call-time, not def-time. And I do not like PEP 3107 neither: its overly complex. If there is a need for Python type checking, I'd suggested to make a special superset which could be used to write compiled extensions as well (Pyrex comes to mind). Regards, Roman P.S. However, I may be wrong. In that case my syntax suggestion would be this: def foo(non_const or []): ... where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone. I have not checked, but I believe it is backward compatible. Anyway, could you summarize both contr-argument and this syntax proposal in the PEP? Chris Rebert wrote:
The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert
Title: Fixing Non-constant Default Arguments
Abstract
This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly.
Motivation
Currently, to write functions using non-constant default arguments, one must use the idiom:
def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function
or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate?
Rationale
Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were:
def foo(bar=<baz>): #code
def foo(bar=new baz): #code
def foo(bar=fresh baz): #code
def foo(bar=separate baz): #code
def foo(bar=another baz): #code
def foo(bar=unique baz): #code
where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below.
Specification
The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter.
Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a
>>> foo() hi >>> CONST="bye" >>> foo() bye
#...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)]
>>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3)
#default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum
def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b
>>> bar() bbbb
#default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c
>>> baz() 2 >>> baz() 3
#...but only when they're required def silly(): print "spam" return 42
def qux(d=silly()): pass
>>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls
#Rule 3 count = 0 def next(): global count count += 1 return count - 1
def frobnicate(g=next(), h=next(), i=next()): print g, h, i
>>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition
Backwards Compatibility
This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:
def foo(bar=mutable): #code
to
def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap
@stateify(mutable) def foo(bar): #code
or
state = mutable def foo(bar=state): #code
or
class Baz(object): def __init__(self): self.state = mutable
def foo(self, bar=self.state): #code
The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used.
In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs.
Reference Implementation
All code of the form:
def foo(bar=some_expr, baz=other_expr): #body
Should act as if it had read (in pseudo-Python):
def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body
where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed.
References
[0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html
[1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6
[2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2
[3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/
[4] Function Annotations http://www.python.org/dev/peps/pep-3107/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
!DSPAM:45bcf8295417644621095!
On 1/29/07, Roman Susi <rnd@onego.ru> wrote:
P.S. However, I may be wrong. In that case my syntax suggestion would be this:
def foo(non_const or []): ...
where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone.
It would also be possible to treat literals (like "[]") as "do it over each time", and more general expressions (like "list()") as they are treated today. Though like Roman, I think this would still be worse than the status quo. -jJ
On 1/29/07, Jim Jewett <jimjjewett@gmail.com> wrote:
On 1/29/07, Roman Susi <rnd@onego.ru> wrote:
P.S. However, I may be wrong. In that case my syntax suggestion would be this:
def foo(non_const or []): ...
where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone.
It would also be possible to treat literals (like "[]") as "do it over each time", and more general expressions (like "list()") as they are treated today.
Though like Roman, I think this would still be worse than the status quo.
Another, more bizarre approach would be to require default arguments be hashable. This would prevent lists from being used, protecting people from mistakes. None of the options given sit right with me though. -- Adam Olsen, aka Rhamphoryncus
Adam Olsen wrote:
[snip] Another, more bizarre approach would be to require default arguments be hashable. This would prevent lists from being used, protecting people from mistakes.
If my PEP ends up being rejected, I would fully support this. It wouldn't solve the more general problem, but at least it would prevent a common newbie mistake. - Chris Rebert
On 1/29/07, Roman Susi <rnd@onego.ru> wrote:
P.S. However, I may be wrong. In that case my syntax suggestion would be this:
def foo(non_const or []): ...
where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone.
It would also be possible to treat literals (like "[]") as "do it over each time", and more general expressions (like "list()") as they are treated today. As discussed in the proto-PEP, there are many cases where non-literal default values need to be re-evaluated at each call. A solution to this
Jim Jewett wrote: problem needs to take this into account. Also, the proto-PEP's proposal tries to have the behavior of default arguments be as uniform as possible and not "magically" decide the correct behavior, especially by using something as arbitrary as what expression is used for the default argument. - Chris Rebert
Chris Rebert wrote:
The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated.
As I see it, the main objection to this is the inversal of the current default semantics: every default argument is treated as if it were mutable, or re-evaluatable more generally. Although mutable default arguments are useful some times and would be nice to have, they are most likely less common than the immutable ones, so the latter should be the default. I'm sure that the python-devs and the BDFL would have thought about it quite a bit when the current semantics were decided, and it's unlikely they'll change their mind now without very good reasons. OTOH, a proposal that leaves the current semantics as is and adds a mechanism to specify default arguments to be evaluated at call-time rather than definition-time would have more chances. Here's a proof-of-concept solution that specifies explicitly the re-evaluatable default expressions as "deferred": import inspect class Deferred(object): def __init__(self, expr): self.expr = expr def eval_deferreds(func): varnames,_,_,defaults = inspect.getargspec(func) num_varnames = len(varnames); num_defaults = len(defaults) def wrapper(*args, **kwds): if len(args) >= num_varnames: # defaults not used here return func(*args,**kwds) f_locals = dict(zip(varnames,args)) used_defaults = min(num_defaults, num_varnames-len(args)) for var,default in zip(varnames[-used_defaults:], defaults[-used_defaults:]): if var in kwds: # passed as keyword argument; don't use the default value = kwds[var] elif not isinstance(default, Deferred): # non re-evaluatable default value = default else: # evaluatable default in f_locals value = eval(default.expr, func.func_globals, f_locals) f_locals[var] = value f_locals.update(kwds) # add any extra keyword arguments return func(**f_locals) return wrapper #======= example ============================== W = 1 # some global @eval_deferreds def f(x, y=Deferred('x**2+W'), z=Deferred('[]')): z.append(x) z.append(y) return z from collections import deque print f(3) # [3,10] W=3; print f(4) # [4,19] print f(4,5) # [4,5] print f(-1, z=deque()) # deque([-1,4]) Regards, George
George Sakkis wrote:
As I see it, the main objection to this is the inversal of the current default semantics: every default argument is treated as if it were mutable, or re-evaluatable more generally. Although mutable default arguments are useful some times and would be nice to have, they are most likely less common than the immutable ones, so the latter should be the default.
Why? Yes, there _might_ be performance issues (which have yet to be demonstrated or deeply speculated upon), but re-evaluating immutable default arguments wouldn't affect a program's correct operation.
I'm sure that the python-devs and the BDFL would have thought about it quite a bit when the current semantics were decided,
...which was probably a while ago. They might reconsider the issue now that some time has passed and they've seen how their decision has worked out. But yes, your analysis is a definite possibility.
and it's unlikely they'll change their mind now without very good reasons.
I hope to provide those reasons in my PEP.
OTOH, a proposal that leaves the current semantics as is and adds a mechanism to specify default arguments to be evaluated at call-time rather than definition-time would have more chances.
My PEP does discuss the possibility of adding new syntax for the new semantics and leaving the old semantics as the default. The final proposal will take into account the community's opinion as to the specifics of how the syntax/semantics ought to be changed.
Here's a proof-of-concept solution that specifies explicitly the re-evaluatable default expressions as "deferred":
import inspect
class Deferred(object): def __init__(self, expr): self.expr = expr
def eval_deferreds(func): varnames,_,_,defaults = inspect.getargspec(func) num_varnames = len(varnames); num_defaults = len(defaults) def wrapper(*args, **kwds): if len(args) >= num_varnames: # defaults not used here return func(*args,**kwds) f_locals = dict(zip(varnames,args)) used_defaults = min(num_defaults, num_varnames-len(args)) for var,default in zip(varnames[-used_defaults:], defaults[-used_defaults:]): if var in kwds: # passed as keyword argument; don't use the default value = kwds[var] elif not isinstance(default, Deferred): # non re-evaluatable default value = default else: # evaluatable default in f_locals value = eval(default.expr, func.func_globals, f_locals) f_locals[var] = value f_locals.update(kwds) # add any extra keyword arguments return func(**f_locals) return wrapper
#======= example ==============================
W = 1 # some global
@eval_deferreds def f(x, y=Deferred('x**2+W'), z=Deferred('[]')): z.append(x) z.append(y) return z
from collections import deque print f(3) # [3,10] W=3; print f(4) # [4,19] print f(4,5) # [4,5] print f(-1, z=deque()) # deque([-1,4])
While that is some pretty nifty/fancy coding, the use of strings for the default values does seem a bit kludgey. However, if my proposal does not end up getting approved, I'll be sure to recommend that some of the great decorators mentioned on this thread get added to the standard library. - Chris Rebert
On 1/29/07, Chris Rebert <cvrebert@gmail.com> wrote:
George Sakkis wrote:
As I see it, the main objection to this is the inversal of the current default semantics: every default argument is treated as if it were mutable, or re-evaluatable more generally. Although mutable default arguments are useful some times and would be nice to have, they are most likely less common than the immutable ones, so the latter should be the default.
Why? Yes, there _might_ be performance issues (which have yet to be demonstrated or deeply speculated upon), but re-evaluating immutable default arguments wouldn't affect a program's correct operation.
If the underlying intent of your proposal -- that all default arguments be re-evaluated with every call -- were to be approved, there would undoubtedly be a serious performance impact. The alternative is an ugly, narrow-use syntax that seeks to eliminate two lines of boilerplate per default argument, boilerplate that can already be replaced with decorators. As for whether these effects have "yet to be demonstrated", as you say, the burden is on you, the PEP author, to investigate and resolve, mitigate or justify any and all performance changes. Collin Winter
Collin Winter wrote:
On 1/29/07, Chris Rebert <cvrebert@gmail.com> wrote: Why? Yes, there _might_ be performance issues (which have yet to be demonstrated or deeply speculated upon), but re-evaluating immutable default arguments wouldn't affect a program's correct operation.
If the underlying intent of your proposal -- that all default arguments be re-evaluated with every call -- were to be approved, there would undoubtedly be a serious performance impact. [snip] As for whether these effects have "yet to be demonstrated", as you say, the burden is on you, the PEP author, to investigate and resolve, mitigate or justify any and all performance changes.
Collin Winter
You do have a point there, it is my responsibility. I therefore shall endeavor to find out how much of a performance hit my suggested change would have. - Chris Rebert
On Tue, 30 Jan 2007 06:40:35 +0100, Chris Rebert <cvrebert@gmail.com> wrote:
George Sakkis wrote:
I'm sure that the python-devs and the BDFL would have thought about it quite a bit when the current semantics were decided,
...which was probably a while ago. They might reconsider the issue now that some time has passed and they've seen how their decision has worked out. But yes, your analysis is a definite possibility.
Just looked it up, and python has had lexical variables since version 2.1, and default arguments since long before that. (forever?) Without lexical variables it's near impossible to implement re-evaluating default arguments, because the variables those default expressions refer to may no longer be available at function call time. So, The BDFL and other python devs didn't really have a choice but to have the default expressions evaluate at definition time (or implement lexical scopes, which is what has happened by now). Jan
Roman Susi wrote:
Hello!
I'd liked to say outright that this bad idea which complicates matters more than provides solutions. Right now it is enough to know that the part from def to ":" is executed at definition time. This is what incremental dynamic semantics is about. So, the suggestion is good only as separated feature, but is IMHO wrong if considered in the language design as a whole.
You're entitled to your own opinion on the PEP.
So things like
def foo(non_const=None): non_const = non_const or []
are good becuase explicitely tell you that the mutable object is to be created at call-time, not def-time.
The 'new' (or similar) keyword (might) indicate the new semantics, or alternatively, 'old' (or a similar) keyword (might) indicate the old semantics. If the new semantics become the default (as the PEP proposes), then this point is moot anyway as it will be explicit by way of the language definition.
And I do not like PEP 3107 neither: its overly complex.
If there is a need for Python type checking, I'd suggested to make a special superset which could be used to write compiled extensions as well (Pyrex comes to mind).
That's not part of my proposal, nor does my PEP depend on that one. I merely mention PEP 3107 when considering compatibility with other 3100-series PEPs.
P.S. However, I may be wrong. In that case my syntax suggestion would be this:
def foo(non_const or []): ...
where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone. I have not checked, but I believe it is backward compatible. Anyway, could you summarize both contr-argument and this syntax proposal in the PEP?
I don't quite understand exactly how this would work and would like more details on it, but once you've explained it, of course I'd be happy to include it in the next draft. - Chris Rebert
Chris Rebert wrote:
Roman Susi wrote:
Hello!
I'd liked to say outright that this bad idea which complicates matters
[skip]
P.S. However, I may be wrong. In that case my syntax suggestion would be this:
def foo(non_const or []): ...
where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone. I have not checked, but I believe it is backward compatible. Anyway, could you summarize both contr-argument and this syntax proposal in the PEP?
I don't quite understand exactly how this would work and would like more details on it, but once you've explained it, of course I'd be happy to include it in the next draft.
Simple. def foo(non_const or []): ... is equivalent to def foo(non_const=None): if non_const is None: none_const = [] ... And this will be as before: def foo(non_const=[]): ... Also, I thing that programmers should not use subtle difference between None and other False values, so something like def foo(non_const=None): non_const = none_const or [] is also valid. Another approach (if you want to pursue the feature) could be complication to name binding protocol. a = [] will be as before, but default value assignment could trigger some extra method. So, you can explicitly regulate your instance reaction to default-value assignment: class MyMutableList: ... def __default__(self, old_default): return old_default.copy() Regards, Roman
- Chris Rebert
Added to PEP draft. Thanks. - Chris Rebert Roman Susi wrote:
Chris Rebert wrote:
Roman Susi wrote:
Hello!
I'd liked to say outright that this bad idea which complicates matters
[skip]
P.S. However, I may be wrong. In that case my syntax suggestion would be this:
def foo(non_const or []): ...
where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone. I have not checked, but I believe it is backward compatible. Anyway, could you summarize both contr-argument and this syntax proposal in the PEP?
I don't quite understand exactly how this would work and would like more details on it, but once you've explained it, of course I'd be happy to include it in the next draft.
Simple.
def foo(non_const or []): ...
is equivalent to
def foo(non_const=None): if non_const is None: none_const = [] ...
And this will be as before:
def foo(non_const=[]): ...
Also, I thing that programmers should not use subtle difference between None and other False values, so something like
def foo(non_const=None): non_const = none_const or []
is also valid.
Another approach (if you want to pursue the feature) could be complication to name binding protocol.
a = []
will be as before, but default value assignment could trigger some extra method. So, you can explicitly regulate your instance reaction to default-value assignment:
class MyMutableList: ... def __default__(self, old_default): return old_default.copy()
Regards, Roman
- Chris Rebert
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
Well, I (obviously) like the idea, but your pep misses some important points (and some not so important). The important one is about the scoping of variables in default expressions. The pep says nothing about them. If I read the pep correctly, any variables in default expressions are handled the same as variables in the body of a function. This means the compiler decides if they should be local, lexical or global. If there is an assignment to a variable, the compiler makes it a local, else it finds the right enclosing scope (lexical or global). In the current python, this works fine:
a = 123 def foo(b=a): a = 2 print a, b
foo() 2 123 a = 42 foo() 2 123
In the pep, the a in the default expression would be handled just like any other a in the function body, which means it wil become a _local_ variable. Calling the function would then result in an UnboundLocalError. Just like this in current python:
a = 123 def foo(b=None): b = a if b==None else b a = 2 print a
foo()
Traceback (most recent call last): File "<pyshell#22>", line 1, in <module> foo() File "<pyshell#21>", line 2, in foo b = a if b==None else b UnboundLocalError: local variable 'a' referenced before assignment The solution, I think, as I wrote in my previous messages, is to have the compiler explicitly make variables in default expressions lexical or global variables. This would still break the foo() in my example above, because you can't assign to a lexical variable, or to a global that isn't declared global. Therefore I think the compiler should distinguish between the a in the default expression and the a in the function body, and treat them as two different variables. You can think about it as if the compiler silently renames one of them (without this rename being visible to python code. AFAIK the bytecode is stack based and closure vars are put in some kind of anonymous cell, which means they both don't actually have a name anyway.) see below for more comments regarding this and other things On Sun, 28 Jan 2007 20:22:44 +0100, Chris Rebert <cvrebert@gmail.com> wrote:
The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert
Title: Fixing Non-constant Default Arguments
Abstract
This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly.
Motivation
Currently, to write functions using non-constant default arguments, one must use the idiom:
def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function
or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2].
Also, I just found out that python's own documentation refers to this with an "Important warning: The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes. ..." (http://docs.python.org/tut/node6.html#SECTION006710000000000000000) this indicates imo that also the python doc writers don't think of the current situation as optimal.
There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate?
Rationale
Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were:
def foo(bar=<baz>): #code
def foo(bar=new baz): #code
def foo(bar=fresh baz): #code
def foo(bar=separate baz): #code
def foo(bar=another baz): #code
def foo(bar=unique baz): #code
where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below.
Specification
The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter.
Include something saying that any variables in a default expression shall be lexical variables, with their scope being the first outer scope that defines a variable with the same name (they should just use the same rules as other lexical/closure variables), and that if the function body defines a local variable with the same name as a variable in a default expression, those variables shall be handled as two separate variables.
- The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter.
Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a
>>> foo() hi >>> CONST="bye" >>> foo() bye
#...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)]
>>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3)
I'm not sure if this can be combined elegantly with what I said about variables being lexical variables. The first argument to ncopies, 'container', is clearly a local variable to ncopies. The 'container' in the second arg default expr should, if my comments above are accepted, be a lexical variable referring to the 'container' in the global scope. The best way to combine the two features seems to be to let 'container' be a local var if any of the preceding args is named 'container', and let it be a lexically scoped variable otherwise. However, I'm not convinced this complexity is worth it and the vars in default expressions shouldn't just always be lexical vars.
#default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum
def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b
>>> bar() bbbb
#default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c
>>> baz() 2 >>> baz() 3
#...but only when they're required def silly(): print "spam" return 42
def qux(d=silly()): pass
>>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls
#Rule 3 count = 0 def next(): global count count += 1 return count - 1
def frobnicate(g=next(), h=next(), i=next()): print g, h, i
>>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition
Backwards Compatibility
This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:
Wow, let's not scare everyone away just yet. This should read: "This change in semantics breaks code which uses mutable default argument expressions and depends on those expressions being evaluated only once, or code that assigns new incompatible values in a parent scope to variables used in default expressions"
def foo(bar=mutable): #code
if 'mutable' is just a single variable, this isn't gonna break, unless the global scope decides to do something like this: def foo(bar=mutable): #code mutable = incompatible_mutable # ... foo()
to
def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap
@stateify(mutable) def foo(bar): #code
or
state = mutable def foo(bar=state): #code
or
class Baz(object): def __init__(self): self.state = mutable
def foo(self, bar=self.state): #code
Minor point: the stateify decorator looks a bit scary to me as it uses three levels of nested functions. (that's inherent to decorators, but still.) Suggest you name the class and global var solutions first, and the decorator as last, just to prevent people from stopping reading the pep and voting '-1' right when they hit the decorator solution.
The changes in this PEP are backwards-compatible with all code whose default argument values are immutable
...or don't depend on being evaluated only once, and don't modify in a parent scope the variables in default expressions in an incompatible way, (hmm, the 'or' and 'and' may need some disambiguation parentheses...)
including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used.
In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs.
Reference Implementation
All code of the form:
def foo(bar=some_expr, baz=other_expr): #body
Should act as if it had read (in pseudo-Python):
def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body
and, if there are any variables occuring in the function body and in some_expr or other_expr, rename those in the function body to something that doesn't name-clash.
where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed.
References
[0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html
[1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6
[2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2
[3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/
[4] Function Annotations http://www.python.org/dev/peps/pep-3107/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
I'm not quite sure if your decision not to include the default-expr-vars-become-lexical part was intentional or not. If it is, can you tell me why you'd want that? Else you can incorporate my comments in the pep. Another minor point: I'm personally not too fond of too many 'shall's close together. It makes me think of lots of bureaucracy and design-by-committee. I think several peps don't have a shall in them, but maybe it is the right language for this one. What's the right language to use in peps? Well, that's about it I think. - Jan
Wow, that's a lot to think about. Yes, the exact nature of the variables in default arguments does need clarification. I'll probably go with something like your proposal, but I need to consider a few things, particularly the 'ncopies' situation. I added a reference to the Python documentation you mentioned. Thanks!
def foo(bar=mutable): #code
if 'mutable' is just a single variable, this isn't gonna break, unless the global scope decides to do something like this:
def foo(bar=mutable): #code
mutable = incompatible_mutable # ... foo()
Actually, I was considering the case where mutable is a constant (e.g. a list), in which case it *will* break since the expr 'mutable' will be re-evaluated at every call, so modifying it won't have the same effect on future calls that it used to. I included your rephrasing under "Backwards Compatibility". I shuffled around the refactorings as you suggested. My decision not to include the default-expr-vars-become-lexical part was completely unintentional, as you can guess from above. I also reworded the sentences using 'shall'. I don't really get why that matters, but what the heck. Thanks for your useful comments and suggestions. - Chris Rebert Jan Kanis wrote:
Well, I (obviously) like the idea, but your pep misses some important points (and some not so important).
The important one is about the scoping of variables in default expressions. The pep says nothing about them. If I read the pep correctly, any variables in default expressions are handled the same as variables in the body of a function. This means the compiler decides if they should be local, lexical or global. If there is an assignment to a variable, the compiler makes it a local, else it finds the right enclosing scope (lexical or global). In the current python, this works fine:
a = 123 def foo(b=a): a = 2 print a, b
foo() 2 123 a = 42 foo() 2 123
In the pep, the a in the default expression would be handled just like any other a in the function body, which means it wil become a _local_ variable. Calling the function would then result in an UnboundLocalError. Just like this in current python:
a = 123 def foo(b=None): b = a if b==None else b a = 2 print a
foo()
Traceback (most recent call last): File "<pyshell#22>", line 1, in <module> foo() File "<pyshell#21>", line 2, in foo b = a if b==None else b UnboundLocalError: local variable 'a' referenced before assignment
The solution, I think, as I wrote in my previous messages, is to have the compiler explicitly make variables in default expressions lexical or global variables. This would still break the foo() in my example above, because you can't assign to a lexical variable, or to a global that isn't declared global. Therefore I think the compiler should distinguish between the a in the default expression and the a in the function body, and treat them as two different variables. You can think about it as if the compiler silently renames one of them (without this rename being visible to python code. AFAIK the bytecode is stack based and closure vars are put in some kind of anonymous cell, which means they both don't actually have a name anyway.)
see below for more comments regarding this and other things
On Sun, 28 Jan 2007 20:22:44 +0100, Chris Rebert <cvrebert@gmail.com> wrote:
The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert
Title: Fixing Non-constant Default Arguments
Abstract
This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly.
Motivation
Currently, to write functions using non-constant default arguments, one must use the idiom:
def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function
or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2].
Also, I just found out that python's own documentation refers to this with an "Important warning: The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes. ..." (http://docs.python.org/tut/node6.html#SECTION006710000000000000000) this indicates imo that also the python doc writers don't think of the current situation as optimal.
There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate?
Rationale
Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were:
def foo(bar=<baz>): #code
def foo(bar=new baz): #code
def foo(bar=fresh baz): #code
def foo(bar=separate baz): #code
def foo(bar=another baz): #code
def foo(bar=unique baz): #code
where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below.
Specification
The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter.
Include something saying that any variables in a default expression shall be lexical variables, with their scope being the first outer scope that defines a variable with the same name (they should just use the same rules as other lexical/closure variables), and that if the function body defines a local variable with the same name as a variable in a default expression, those variables shall be handled as two separate variables.
- The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter.
Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a
>>> foo() hi >>> CONST="bye" >>> foo() bye
#...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)]
>>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3)
I'm not sure if this can be combined elegantly with what I said about variables being lexical variables. The first argument to ncopies, 'container', is clearly a local variable to ncopies. The 'container' in the second arg default expr should, if my comments above are accepted, be a lexical variable referring to the 'container' in the global scope. The best way to combine the two features seems to be to let 'container' be a local var if any of the preceding args is named 'container', and let it be a lexically scoped variable otherwise. However, I'm not convinced this complexity is worth it and the vars in default expressions shouldn't just always be lexical vars.
#default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum
def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b
>>> bar() bbbb
#default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c
>>> baz() 2 >>> baz() 3
#...but only when they're required def silly(): print "spam" return 42
def qux(d=silly()): pass
>>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls
#Rule 3 count = 0 def next(): global count count += 1 return count - 1
def frobnicate(g=next(), h=next(), i=next()): print g, h, i
>>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition
Backwards Compatibility
This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:
Wow, let's not scare everyone away just yet. This should read: "This change in semantics breaks code which uses mutable default argument expressions and depends on those expressions being evaluated only once, or code that assigns new incompatible values in a parent scope to variables used in default expressions"
def foo(bar=mutable): #code
if 'mutable' is just a single variable, this isn't gonna break, unless the global scope decides to do something like this:
def foo(bar=mutable): #code
mutable = incompatible_mutable # ... foo()
to
def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap
@stateify(mutable) def foo(bar): #code
or
state = mutable def foo(bar=state): #code
or
class Baz(object): def __init__(self): self.state = mutable
def foo(self, bar=self.state): #code
Minor point: the stateify decorator looks a bit scary to me as it uses three levels of nested functions. (that's inherent to decorators, but still.) Suggest you name the class and global var solutions first, and the decorator as last, just to prevent people from stopping reading the pep and voting '-1' right when they hit the decorator solution.
The changes in this PEP are backwards-compatible with all code whose default argument values are immutable
...or don't depend on being evaluated only once, and don't modify in a parent scope the variables in default expressions in an incompatible way,
(hmm, the 'or' and 'and' may need some disambiguation parentheses...)
including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used.
In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs.
Reference Implementation
All code of the form:
def foo(bar=some_expr, baz=other_expr): #body
Should act as if it had read (in pseudo-Python):
def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body
and, if there are any variables occuring in the function body and in some_expr or other_expr, rename those in the function body to something that doesn't name-clash.
where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed.
References
[0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html
[1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6
[2] When Pythons Attack
http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2
[3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/
[4] Function Annotations http://www.python.org/dev/peps/pep-3107/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
I'm not quite sure if your decision not to include the default-expr-vars-become-lexical part was intentional or not. If it is, can you tell me why you'd want that? Else you can incorporate my comments in the pep. Another minor point: I'm personally not too fond of too many 'shall's close together. It makes me think of lots of bureaucracy and design-by-committee. I think several peps don't have a shall in them, but maybe it is the right language for this one. What's the right language to use in peps?
Well, that's about it I think.
- Jan
On Tue, 30 Jan 2007 08:09:37 +0100, Chris Rebert <cvrebert@gmail.com> wrote:
I also reworded the sentences using 'shall'. I don't really get why that matters, but what the heck.
Well, it doesn't. Just makes it nicer to read. (As long as the text is still clear, that is.)
Jan Kanis wrote:
a = 123 def foo(b=a): a = 2 print a, b foo() 2 123 a = 42 foo() 2 123 In the pep, the a in the default expression would be handled just like any other a in the function body, which means it wil become a _local_ variable. Calling the function would then result in an UnboundLocalError. Just like this in current python: a = 123 def foo(b=None): b = a if b==None else b a = 2 print a foo() Traceback (most recent call last): File "<pyshell#22>", line 1, in <module> foo() File "<pyshell#21>", line 2, in foo b = a if b==None else b UnboundLocalError: local variable 'a' referenced before assignment The solution, I think, as I wrote in my previous messages, is to have
Well, I (obviously) like the idea, but your pep misses some important points (and some not so important). The important one is about the scoping of variables in default expressions. The pep says nothing about them. If I read the pep correctly, any variables in default expressions are handled the same as variables in the body of a function. This means the compiler decides if they should be local, lexical or global. If there is an assignment to a variable, the compiler makes it a local, else it finds the right enclosing scope (lexical or global). In the current python, this works fine: the compiler explicitly make variables in default expressions lexical or global variables. This would still break the foo() in my example above, because you can't assign to a lexical variable, or to a global that isn't declared global. Therefore I think the compiler should distinguish between the a in the default expression and the a in the function body, and treat them as two different variables. You can think about it as if the compiler silently renames one of them (without this rename being visible to python code. AFAIK the bytecode is stack based and closure vars are put in some kind of anonymous cell, which means they both don't actually have a name anyway.)
Also, I've been thinking about the part that I suggested about having python distinguish between lexically scoped variables in default expressions and local variables that may have the same name. To clarify my proposal, a demonstration: Current python: a = 4 b = 5 def foo(x = [b]*a): x = copy.deepcopy(x) # one possible workaround to have the same list on every call b = 'bla' # no name clash of this b and b in default arg expr x += [b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] proposed semantics: a = 4 b = 5 def foo(x = [b]*a): # deepcopy() no longer nescessary b = 'bla' # no name clash of this b and b in default arg expr, the compiler distinguishes them x += [b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] This would work the same as this code in the current python. Imagine it as the compiler silently transforming the above code to this code: a = 4 b = 5 def foo(x = _undefined): if x == _undefined: x = [b]*a _b = 'bla' x += [_b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] note: the use of deepcopy assumes that whatever is passed to foo is deepcopyable. With the new semantics this function can be used without these kind of assumptions. However, I've been thinking about weather it is a good idea to have python distinguish between the local and lexical 'b' in the above code, and do the what you can think of as implicit renaming. Making the distinction makes the whole proposal more backward compatible, however I think that if backward compatibility were not an issue, it would be best not to make the distinction and just have the programmer choose a different name for the local variable. This would improve readability as there are no two different variables with the same name in the same piece of code, and decrease language complexity as there is no rule that's best explained in terms of variable name rewriting. Not making the distinction does cause the default arguments proposal to be less backward compatible, but I don't know if that is really a big problem. I assume this pep wil be a py3k pep anyway, so there are going to be incompatibilities anyway. Also, the compiler is perfectly able to see at compile time if a variable used in a default expression and as a local, so it is able to raise a very descriptive error. All in all, I think this part of my proposal is not such a good idea, unless there's going to be some kind of transitional implementation of this pep in py 2.x meta-note: I hope it is clear what I'm trying to say. Basically, I'm clarifying, arguing against, and retracting a part of my own proposal I made in an earlier mail. - Jan
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
Yes, it is an ugly gotcha, but so is the alternative. If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics. In a dynamic language, it makes perfect sense to reevaluate the entire call signature with each call -- but in almost any language, it makes perfect sense for the signature to be a constant. Usually, it doesn't matter which you assume, which is why this problem doesn't get ironed out in the first few minutes of writing python.
There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables.
I disagree. This is particularly wrong for someone coming from a functional background. A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java. You see the problem with globals, but decorators are in some sense worse -- a function cannot see its own decorations. At best, it can *assume* that repeating its own name (despite Don't Repeat Yourself) will get another reference to self, but this isn't always true. Programmers used to creating functions outside of toplevel (or class-level) will be more aware of this, and see the suggestion as an indication that python is inherently buggy.
def foo(bar=new baz): #code
This would be less bad. That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.
Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a
This would work if there were any easy way to create a new scope. In Lisp, it makes sense. In python, it would probably be better to just find a way for functions to refer to their own decorations reliably.
Backwards Compatibility
This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:
def foo(bar=mutable): #code
to
[a decorator option uses 3 levels of nested functions, which aren't even generic enough for reuse, unless you give up introspection.] No. It could be done with a (more complex) decorator, if that decorator came with the stdlib (probably in functools), but that is heavy backwards-incompatibility to change a corner case (most defaults aren't mutable) in a taste-dependent manner. -jJ
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
Yes, it is an ugly gotcha, but so is the alternative.
If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics.
Anyone (ab)using the current semantics with the above construct is or ought to be aware that such usage is unpythonic. Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor. Additionally, I see making the opposite mistake as usually only maybe causing a performance problem, as opposed to causing the program to work incorrectly. As discussed in my proposal, this might be premature optimization. If it is a serious issue (which hasn't been indicated by the discussion so far), we can add syntax for the old/new semantics, which I also mentioned in the proposal.
There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables.
I disagree. This is particularly wrong for someone coming from a functional background.
I assume you disagree with the "purpose is much better served by decorators, classes, or local variables" part as opposed to the "default mutable arguments with current semantics have few good uses" part. Please correct me if I'm in error.
A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java.
I'm sorry, but when you have a function sharing state between calls, that just screams to me that you should make it a method of an object so you don't have to store the state in some roundabout way, such as in mutable default arguments. If performance is your concern, the decorator version might perform better (I don't really know), or in the extreme case, you could use a global variable, which is definitely faster. Speed and elegance often come in inverse proportions. Also, I've revised the refactoring in question so that you no longer need to instanciate the class, which at least makes it marginally better.
You see the problem with globals, but decorators are in some sense worse -- a function cannot see its own decorations. At best, it can *assume* that repeating its own name (despite Don't Repeat Yourself) will get another reference to self, but this isn't always true.
I really don't see how the decorator in the PEP is any worse than other decorators in this regard. The problem you describe currently applies to all decorated functions, though functools.wraps might help mitigate this situation.
Programmers used to creating functions outside of toplevel (or class-level) will be more aware of this, and see the suggestion as an indication that python is inherently buggy.
def foo(bar=new baz): #code
This would be less bad.
That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.
This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x is None: x=expr' idiom in that it expresses one's intent more clearly. Also, this would at least be a prettier way to spell the idiom even if the reason still needed explaining. Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons. I think I'll add this to the PEP.
Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a
This would work if there were any easy way to create a new scope. In Lisp, it makes sense. In python, it would probably be better to just find a way for functions to refer to their own decorations reliably.
This is outside of the scope of my PEP. However, the below improvement should help and you could always use one of the other refactorings to work around this issue.
Backwards Compatibility
This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:
def foo(bar=mutable): #code
to
[a decorator option uses 3 levels of nested functions, which aren't even generic enough for reuse, unless you give up introspection.]
No. It could be done with a (more complex) decorator, if that decorator came with the stdlib (probably in functools)
Agreed, the decorator could be better. I've just enhanced it using functools.wraps, which should help with the introspection issues you raise. Other improvements to the refactoring code in the PEP are welcomed. - Chris Rebert
On 1/30/07, Chris Rebert <cvrebert@gmail.com> wrote:
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
Yes, it is an ugly gotcha, but so is the alternative.
If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics.
Anyone (ab)using the current semantics with the above construct is or ought to be aware that such usage is unpythonic.
Are you worried about naive programmers or not?
Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor.
I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process.
Additionally, I see making the opposite mistake as usually only maybe causing a performance problem, as opposed to causing the program to work incorrectly.
That is true only when the argument is a cache. Generally, resetting the storage will make the program work incorrectly, but the program won't fail on the first data tested -- which means the bug is less likely to be caught.
There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables.
I disagree. This is particularly wrong for someone coming from a functional background.
I assume you disagree with the "purpose is much better served by decorators, classes, or local variables" part as opposed to the "default mutable arguments with current semantics have few good uses" part. Please correct me if I'm in error.
I disagree with both. There are many uses for preserving function state between calls. C uses static local variables. Object Oriented languages often use objects.
A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java.
I'm sorry, but when you have a function sharing state between calls, that just screams to me that you should make it a method of an object
That's because you drank the OO koolaid. Python tries not to enforce any particular programming style. Its object model happens to be pretty good, but there are still times when OO isn't the answer.
so you don't have to store the state in some roundabout way, such as in mutable default arguments. If performance is your concern, the decorator version might perform better (I don't really know), or in the extreme case, you could use a global variable, which is definitely faster.
I think there is something wrong with your benchmark. The default argument is accessible straight from the function object itself; even a cell variable shouldn't be that fast, let alone global variables. That said, speed is *not* my concern -- code size is. If I have to add several lines of boilerplate, the code becomes much less readable. That is roughly the same justification you have for not liking the "if arg is None: arg=foo()" idiom. The difference is that you're suggesting adding far more than a line or two.
You see the problem with globals, but decorators are in some sense worse -- a function cannot see its own decorations. At best, it can *assume* that repeating its own name (despite Don't Repeat Yourself) will get another reference to self, but this isn't always true.
Programmers used to creating functions outside of toplevel (or class-level) will be more aware of this, and see the suggestion as an indication that python is inherently buggy.
I really don't see how the decorator in the PEP is any worse than other decorators in this regard. The problem you describe currently applies to all decorated functions, though functools.wraps might help mitigate this situation.
Either you effectively hide the real function so as to create a lexical closure (plenty of extra work, both mentally and computationally) or you accept buggy code. The bug possibility isn't inherent to decorations; only to functions reading their *own* decorations.
Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons.
Using a mutable default is almost never for optimization reasons. -jJ
On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett <jimjjewett@gmail.com> wrote:
On 1/30/07, Chris Rebert <cvrebert@gmail.com> wrote:
Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor.
I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process.
Well, it seems that numbers are gonna be needed anyway to convince people. But that's certainly possible. What exactly did you search for, and what did you find? I did some preliminary grepping through the standardlib myself. regex = "def.*\(.*=.*\):", searching through all .py files in /Lib: 2455 matches in total in 474 files out of 998 I manually looked through the first bunch of files, those contained 145 matches. (the first files sorted alphabetically up until and including cookielib.py) By far the most of them had literals as default value. A huge amount of them used the =None idiom, I estimate perhaps one third. 10 matches used variables but will not break under the pep's semantics. I found 1 (one) file that used the current semantics to keep state between calls, in two inner functions. This can easily be changed. So, that leaves 145-12=133 uses of default values that are just constants. If I have time and figure out the right regexes I'll try and come up with some more numbers on the entire stdlib, and the ammount of uses of =None. - Jan
On 1/30/07, Jan Kanis <jan.kanis@phil.uu.nl> wrote:
On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett <jimjjewett@gmail.com> wrote:
On 1/30/07, Chris Rebert <cvrebert@gmail.com> wrote:
Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor.
I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process.
But that's certainly possible. What exactly did you search for, and what did you find?
In idle, edit menu, find in files, regex 'def.*=(\\[|\\{)(\\]|\\})' In other words, it caught "={}" or "=[]" on a def line. It did not catch other (possibly mutable) default args, nor did it catch multi-line definitions, nor did it even catch extra whitespace. I didn't notice any false positives where the "={}" was really in a comment. I didn't even look at defaults of None (let alone other singletons) to see if they were part of this idiom. They mostly seemed to be passing state, but the __init__ methods in particular may have been using a class-level variable by accident. (such as the cnf argument to Tk widgets.) I'll try to check more closely later. -jJ
Apologies in advance for the lateness of my reply. I missed a few emails in all the shuffle. Jim Jewett wrote:
On 1/30/07, Chris Rebert <cvrebert@gmail.com> wrote:
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
Naive programmers desiring mutable default arguments often make the mistake of writing the following:
def foo(mutable=some_expr_producing_mutable): #rest of function
Yes, it is an ugly gotcha, but so is the alternative. If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics.
Anyone (ab)using the current semantics with the above construct is or ought to be aware that such usage is unpythonic.
Are you worried about naive programmers or not? [snip]
Additionally, I see making the opposite mistake as usually only maybe causing a performance problem, as opposed to causing the program to work incorrectly.
That is true only when the argument is a cache. Generally, resetting the storage will make the program work incorrectly, but the program won't fail on the first data tested -- which means the bug is less likely to be caught.
I am worried about naive programmers. It seems that the opposite mistake (thinking arguments will be evaluated only once, at def-time, rather than at each call) won't have non-performance related effects a good bit of the time. For many of the remaining cases, the person probably is/should be using a constant instead of a default argument. Yes, there will be some cases where badness could happen. However, (1) this is intended for Py3k, so there will already be a bunch of changes for newbies to learn, (2) the chance of weirdness happening is a good bit lower compared to the current state of mutable default arguments.
Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor.
I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process.
I have a script to gather statistics on default argument usage in the standard library 90% done. Its results will be included in the next PEP draft.
There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or(though less preferred) global variables.
I disagree. This is particularly wrong for someone coming from a functional background.
I assume you disagree with the "purpose is much better served by decorators, classes, or local variables" part as opposed to the "default mutable arguments with current semantics have few good uses" part. Please correct me if I'm in error.
I disagree with both. There are many uses for preserving function state between calls. C uses static local variables. Object Oriented languages often use objects.
First, I was referring to using default args for caching being evil, not mutable default arguments in general. Just wanted to clarify in case you misinterpreted me. True, there are many uses for preserving function state between calls, but how much better/clearer is doing so using default arguments as opposed to objects or other approaches? It's an aesthetic decision. IMHO, the other ways seem better. YMMV.
A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java.
I'm sorry, but when you have a function sharing state between calls, that just screams to me that you should make it a method of an object
That's because you drank the OO koolaid. Python tries not to enforce any particular programming style. Its object model happens to be pretty good, but there are still times when OO isn't the answer.
Well, it's a trade-off: support procedural/functional programming a bit more, or make default arguments a bit more intuitive and obsolete the 'foo=None; if foo is None' idiom.
If performance is your concern, the decorator version might perform better (I don't really know), or in the extreme case, you could use a global variable, which is definitely faster.
I think there is something wrong with your benchmark.
Sorry, yeah, you're right. I was being a little fast and loose. But the point was that there are (at the price of some elegance) ways to optimize such situations even more.
That said, speed is *not* my concern -- code size is. If I have to add several lines of boilerplate, the code becomes much less readable. That is roughly the same justification you have for not liking the "if arg is None: arg=foo()" idiom. The difference is that you're suggesting adding far more than a line or two.
True, however I'm willing to bet that the boilerplate for evaluating default arguments only once won't be needed nearly as much, as such situations requiring such behavior is not too common. The statistics should help show exactly how much mutable default arguments are exploited in the wild.
Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons.
Using a mutable default is almost never for optimization reasons.
Actually, I was more thinking it might be used to stop immutable default arguments from being re-evaluated multiple times, if it turns out that re-evaluations really do have major effects on performance. Now that I think about it, I guess it could be used to get the old behavior for mutable default arguments. - Chris Rebert
Chris Rebert wrote:
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
/skip/
def foo(bar=new baz): #code
This would be less bad.
That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.
This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x
But if you are concerned with the current None, there could be some other, new False value serving the same need, like: def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code or even: def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
is None: x=expr' idiom in that it expresses one's intent more clearly. Also, this would at least be a prettier way to spell the idiom even if the reason still needed explaining. Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons. I think I'll add this to the PEP.
Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a
This would work if there were any easy way to create a new scope. In Lisp, it makes sense. In python, it would probably be better to just find a way for functions to refer to their own decorations reliably.
This is outside of the scope of my PEP. However, the below improvement should help and you could always use one of the other refactorings to work around this issue.
Backwards Compatibility
This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:
def foo(bar=mutable): #code
to
[a decorator option uses 3 levels of nested functions, which aren't even generic enough for reuse, unless you give up introspection.]
No. It could be done with a (more complex) decorator, if that decorator came with the stdlib (probably in functools)
Agreed, the decorator could be better. I've just enhanced it using functools.wraps, which should help with the introspection issues you raise. Other improvements to the refactoring code in the PEP are welcomed.
- Chris Rebert
Roman Susi schrieb:
Chris Rebert wrote:
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
/skip/
def foo(bar=new baz): #code
This would be less bad.
That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.
This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x
But if you are concerned with the current None, there could be some other, new False value serving the same need, like:
def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code
or even:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
why not even def foo(x, y, z, bar=..., qux=...): if bar is Ellipsis: # foo ;) Georg
Roman Susi <rnd@onego.ru> wrote:
def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code
With the proper definition of Missing, the above is fine, and is more or less creating a new 'None' value.
or even:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible. - Josiah
On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson@uci.edu> wrote:
Roman Susi <rnd@onego.ru> wrote:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible.
- Josiah
As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change) - Jan
"Jan Kanis" <jan.kanis@phil.uu.nl> wrote:
On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson@uci.edu> wrote:
Roman Susi <rnd@onego.ru> wrote:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible.
- Josiah
As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change)
Fowards compatible then. That is to say, writing for the new proposed system will break compatibility with older Pythons. On the other hand, using currently-available language syntax and semantics is compatible among the entire range of Pythons available today. - Josiah
Josiah Carlson wrote:
"Jan Kanis" <jan.kanis@phil.uu.nl> wrote:
On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson@uci.edu> wrote:
Roman Susi <rnd@onego.ru> wrote:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible.
- Josiah
As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change)
Fowards compatible then. That is to say, writing for the new proposed system will break compatibility with older Pythons. On the other hand, using currently-available language syntax and semantics is compatible among the entire range of Pythons available today.
From wikipedia:
"... a product is said to be backward compatible (or downward compatible) when it is able to take the place of an older product, by interoperating with other products that were designed for the older product." "Forward compatibility (sometimes confused with extensibility) is the ability of a system to accept input intended for later versions of itself." So, the right term is that Python 3000 will be in the syntactic aspect we discuss backward compatible because it will accept old syntax too. Regards, Roman
- Josiah
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
!DSPAM:45bff1845411635716952!
On Wed, 31 Jan 2007 09:04:33 +0100, Roman Susi <rnd@onego.ru> wrote:
Josiah Carlson wrote:
"Jan Kanis" <jan.kanis@phil.uu.nl> wrote:
On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson@uci.edu> wrote:
Roman Susi <rnd@onego.ru> wrote:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible.
- Josiah
As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change)
Fowards compatible then. That is to say, writing for the new proposed system will break compatibility with older Pythons. On the other hand, using currently-available language syntax and semantics is compatible among the entire range of Pythons available today.
From wikipedia:
"... a product is said to be backward compatible (or downward compatible) when it is able to take the place of an older product, by interoperating with other products that were designed for the older product."
"Forward compatibility (sometimes confused with extensibility) is the ability of a system to accept input intended for later versions of itself."
So, the right term is that Python 3000 will be in the syntactic aspect we discuss backward compatible because it will accept old syntax too.
Regards, Roman
Forward compatible?? There are exactly zero people on python-dev who care about that. What people care about is that a piece of python code written at the time of 2.4 will still run the same on python 2.5. That's backward compatibility. Nobody cares that a piece of python code using syntax constructs that were new in 2.5 won't run on python 2.4. If you want that, then don't use the new constructs. New syntax constructs get added to the language on just about every major revision, no python major version is forward compatible with a higher major version. Python 3.0 is specifically created to allow the backward incompatible changes that people deem nescessary, so neither forward nor backward compatibility matter in that respect. (well, except that it shouldn't become a superhuman task to convert 2.x code to run on python 3.0) - Jan
"Jan Kanis" <jan.kanis@phil.uu.nl> wrote:
Forward compatible?? There are exactly zero people on python-dev who care about that. What people care about is that a piece of python code written at the time of 2.4 will still run the same on python 2.5. That's backward compatibility. Nobody cares that a piece of python code using syntax constructs that were new in 2.5 won't run on python 2.4. If you want that, then don't use the new constructs. New syntax constructs get added to the language on just about every major revision, no python major version is forward compatible with a higher major version.
There are many developers who concern themselves with writing code that will work on version x and x+1 of Python. There are also people who like to use "bleeding edge" features while also using an older Python. If there weren't, then 'from __future__ import generators' wouldn't have existed in Python 2.2, 'from __future__ import floor_division' wouldn't exist in Python 2.3+, 'from __future__ import nested_scopes' wouldn't have existed in Python 2.1 or 2.2 (I can't remember), etc. Yes, there is new syntax with each later Python that won't work with previous Pythons, but with the vast majority of syntax changes that have come down (including the with statement in 2.5), the semantics of the new syntax are defined by using the previously existing language. In this case, the 'def foo(x=):' has a translation into current Python, but it turns out that the translation is *exactly what you should be doing today* if you care about correctness.
Python 3.0 is specifically created to allow the backward incompatible changes that people deem nescessary, so neither forward nor backward compatibility matter in that respect. (well, except that it shouldn't become a superhuman task to convert 2.x code to run on python 3.0)
The reason I concerned myself with this in regards to the 'def foo(x=)' proposal is because the ultimate point of this is to make writing code *less* buggy. However, code can be *less* buggy today through the proper application of an if statement or conditional expression. Yes, Py3k does allow for incompatible syntax. But the syntax changes aren't going to be "becase we can", they are going to be "because this way is better". Leaving out an argument isn't better (in the case of the 'def foo(x=):' syntax); it's ambiguous at best, confusing and worthless at worst. We can argue all day about writing compatible code, breaking compatibility, etc., but the fact remains: no developer who has any pull in python-3000 or python-dev has come out in favor of changing default argument semantics in this thread. - Josiah
Roman Susi wrote:
Chris Rebert wrote:
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
/skip/
def foo(bar=new baz): #code
This would be less bad.
That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.
This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x
But if you are concerned with the current None, there could be some other, new False value serving the same need, like:
def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code
or even:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
Those examples are only slightly better than using None. The convention of using None to indicate use of a mutable/non-constant default value is about as clear as your examples. The point wasn't that None wasn't descriptive/specific enough, but rather why have to write the 'bar=None' and then the 'if bar is None' when 'bar=[]' could be sufficient? Also, this indicates exactly what the default value is, as opposed to None/Missing, which is just an opaque indicator for 'some mutable/non-constant default value'. Thus, I see no need for 'Missing' or a similar constant. - Chris Rebert
Chris Rebert wrote:
Roman Susi wrote:
Chris Rebert wrote:
Jim Jewett wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
/skip/
def foo(bar=new baz): #code
This would be less bad.
That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.
This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x
But if you are concerned with the current None, there could be some other, new False value serving the same need, like:
def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code
or even:
def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code
at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.
Those examples are only slightly better than using None. The convention of using None to indicate use of a mutable/non-constant default value is about as clear as your examples.
The point wasn't that None wasn't descriptive/specific enough, but rather why have to write the 'bar=None' and then the 'if bar is None' when 'bar=[]' could be sufficient? Also, this indicates exactly what the default value is, as opposed to None/Missing, which is just an opaque indicator for 'some mutable/non-constant default value'.
Thus, I see no need for 'Missing' or a similar constant.
And I think it is overkill to add new syntax there. For example, if the function or method is near the external (human or network) interface border it is good to check ALL arguments anyway. So the extra if baz is Missing: baz = [] doesn't spoil the impression: def foo(x, y, z, baz=, qux=): ... if baz is Missing: baz = [] ... if type(baz) != type([]) or len(baz) > 1000 or min(baz) < 0 or ...: raise ... #code What I'd wanted to say is that =None or =Missing is not just "indicator for 'some mutable/non-constant default value'." but may also mean that the argument is subject to further constraints and thus funtion code has checks for that. So, instead of having clear indicator of missing argument we have some fresh instance of a list or dict. Also, in many situation another approach is used nowadays: def foo(x, y, z, **args): ... baz = args.get("baz", []) qux = args.get("qux", {}) Which has the effect proto-PEP author would like to achieve. Of course, no automatic tools are going to make type checks with that helpful function signature. Also, while its offtopic in this post, I'd really liked Python 3000 to have more unification for dict vs attribute access because there are use cases (Storage in web.py, Namespace in mxTools - I am sure many other projects have similar things) which single purpose is to have both dict and attribute syntax in one data object interchangeably: def foo(**args): ... baz = args.baz qux = args["qux"] Regards, Roman
- Chris Rebert
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote:
Rationale [snip] There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were:
def foo(bar=<baz>): #code
def foo(bar=new baz): #code
def foo(bar=fresh baz): #code
def foo(bar=separate baz): #code
def foo(bar=another baz): #code
def foo(bar=unique baz): #code
where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics.
Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have. Also, you haven't talked at all about how all this might be accomplished. You say
Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time.
but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments? Collin Winter
Collin Winter wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.
One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!), and doesn't allow you to refer to other arguments in determining the default value (think 'self' as a hint for how this could be useful). Also, I just think it's plain nicer/prettier if decorators don't have to be used for this.
Also, you haven't talked at all about how all this might be accomplished. You say
Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time.
but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments?
The "Reference Implementation" section of the PEP discusses this. I don't personally know Python's internals, but I imagine this proposal would just change how default arguments are compiled and might entail some changes to the interpreter's argument-processing machinery. As Jan Kanis pointed out, the PEP does need to elaborate on exactly what kind of variables the default arguments use/are. I'm working on revising this in the next draft. - Chris Rebert
On 1/29/07, Chris Rebert <cvrebert@gmail.com> wrote:
Collin Winter wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.
One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work.
Tomer wasn't proposing that any of those decorators be included in the standard library. Those cases where one of decorators isn't sufficient as written? Modify it, use it in your code, post it to the cookbook. Not every n-line function should be turned into syntax.
His second decorator involves using lambdas (ick!), and doesn't allow you to refer to other arguments in determining the default value (think 'self' as a hint for how this could be useful).
You can't do that as things are, so that's not a strike against the decorator-based approach.
Also, you haven't talked at all about how all this might be accomplished. You say
Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time.
but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments?
The "Reference Implementation" section of the PEP discusses this. I don't personally know Python's internals, but I imagine this proposal would just change how default arguments are compiled and might entail some changes to the interpreter's argument-processing machinery.
As you work on this, think about how you'll explain the new semantics in the documentation. Rewriting http://docs.python.org/ref/calls.html and http://docs.python.org/ref/function.html -- paying particular attention to paragraphs 2-5 in the former -- would be a good start. Collin Winter
Collin Winter wrote:
On 1/29/07, Chris Rebert <cvrebert@gmail.com> wrote:
Collin Winter wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.
One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work.
Tomer wasn't proposing that any of those decorators be included in the standard library. Those cases where one of decorators isn't sufficient as written? Modify it, use it in your code, post it to the cookbook. Not every n-line function should be turned into syntax.
Right, but for something used so frequently, should you really have to look through N decorator recipes, figure out which one you want, and then modify it, as opposed to a change in semantics that obviates the need for such decorators altogether?
His second decorator involves using lambdas (ick!), and doesn't allow you to refer to other arguments in determining the default value (think 'self' as a hint for how this could be useful).
You can't do that as things are, so that's not a strike against the decorator-based approach.
Valid point. It's actually more pro-reevaluation than it is anti-decorator.
but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments?
The "Reference Implementation" section of the PEP discusses this. I don't personally know Python's internals, but I imagine this proposal would just change how default arguments are compiled and might entail some changes to the interpreter's argument-processing machinery.
As you work on this, think about how you'll explain the new semantics in the documentation. Rewriting http://docs.python.org/ref/calls.html and http://docs.python.org/ref/function.html -- paying particular attention to paragraphs 2-5 in the former -- would be a good start.
Sweet! Thanks for the link. - Chris Rebert
Chris Rebert <cvrebert@gmail.com> wrote:
Collin Winter wrote:
Tomer wasn't proposing that any of those decorators be included in the standard library. Those cases where one of decorators isn't sufficient as written? Modify it, use it in your code, post it to the cookbook. Not every n-line function should be turned into syntax.
Right, but for something used so frequently, should you really have to look through N decorator recipes, figure out which one you want, and then modify it, as opposed to a change in semantics that obviates the need for such decorators altogether?
Ahh, but the decorator recipes are more or less examples of how one could go about removing the 1 additional if statement/conditional expression. Obviously you don't *need* to look through the decorators, because the 1 line if statement or conditional expression solves them all! Really, you could think of the decorator variants as anti-patterns, because they seek to complicate what is trivial to solve, which isn't an issue in real world code. - Josiah
Chris Rebert <cvrebert@gmail.com> wrote:
Collin Winter wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.
One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!),
[snip] Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*. I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics... def append_10(lst=[]): lst.append(10) return lst I would presumably change it to... def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it). I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use. - Josiah
Josiah Carlson wrote:
Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*.
No, it's just that why use several different decorators when a slight change in semantics renders them all obsolete? Why have to use a decorator/lambda/conditional and remember when to use which? I'm trying to generalize the pattern of "reevaluate/copy default values each time they're required" into new behavior for default arguments. Indeed, as I stated in another email (no idea which one, there's been too many), I wholeheartedly support adding some of those decorators to the standard library should my PEP get rejected.
I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics...
def append_10(lst=[]): lst.append(10) return lst
I would presumably change it to...
def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst
Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it).
Nowhere has it been proposed to forbid passing None as an argument. The basic core of the proposal is to have Python compile the code such that the boilerplate is either removed entirely (if the new semantics are the default) by no longer being necessary, or dramatically shortened via the addition of a new keyword (which would indicate the new semantics). Yes, you can manually transform your code to use the idiom you mention. But why should you have to? Why not have Python do the heavy-lifting for you? As stated in the "Reference Implementation" section of the PEP, the 'translation' provided is not intended to be a literal translation of the code at compile-time, but rather explain how Python's argument-handling machinery should be changed. The 'translation' looks similar to the existing idiom because that's what it aims to replace. But note that it's *Python* that's doing this 'translation', not the programmer! That's the gain! The programmer no longer needs to write the boilerplate.
I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use.
I'm not here to argue that people shouldn't be made to understand the difference between mutable and immutable values. They definitely should know the difference. I'm just advocating a language change to make certain kinds of functions less verbose. If you're worried about losing this 'teaching tool' (which another respondent was worried about), it will still exist in the form of: x = [[0]*4]*4 x[3][1] = 7 x[0][1] == 7 #TRUE! - Chris Rebert
Chris Rebert <cvrebert@gmail.com> wrote:
Josiah Carlson wrote:
Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*.
No, it's just that why use several different decorators when a slight change in semantics renders them all obsolete? Why have to use a decorator/lambda/conditional and remember when to use which? I'm trying to generalize the pattern of "reevaluate/copy default values each time they're required" into new behavior for default arguments. Indeed, as I stated in another email (no idea which one, there's been too many), I wholeheartedly support adding some of those decorators to the standard library should my PEP get rejected.
The conditional works in all cases. The lambda works in all cases. The set of decorators seems to need to be tuned depending on the situation. Me, I would just use a conditional or even a plain if statement, as it is already sufficiently universal to handle every case. Discussions about 'but then I can't mask global names with local names' are going to fall on deaf ears; it's seen as poor form, and discussing what one can do using poor Python form is not topical.
I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics...
def append_10(lst=[]): lst.append(10) return lst
I would presumably change it to...
def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst
Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it).
Nowhere has it been proposed to forbid passing None as an argument.
And I never claimed as much. There was a previous post by someone (I can't remember who offered it), saying that replacing 'lst=[]' with 'lst=None' would make it so that a user couldn't signal *something special* with a 'None' argument. I was trying to get across that such reasoning doesn't make sense in this particular context.
The basic core of the proposal is to have Python compile the code such that the boilerplate is either removed entirely (if the new semantics are the default) by no longer being necessary, or dramatically shortened via the addition of a new keyword (which would indicate the new semantics). Yes, you can manually transform your code to use the idiom you mention. But why should you have to? Why not have Python do the heavy-lifting for you?
Heavy lifting? One line change and one line addition is heavy lifting? There has got to be an episode of Monty Python embedded in this thread somewhere, because I can't help laughing at your claim.
I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use.
I'm not here to argue that people shouldn't be made to understand the difference between mutable and immutable values. They definitely should know the difference. I'm just advocating a language change to make certain kinds of functions less verbose.
My point is that the change results in *certain* functions being trivially less verbose. I've spent more time working around re's call semantics than I have dealing with (incorrect assumptions about Python's) default argument semantics. As I don't see a need to change re, I don't see a need to change default argument semantics. - Josiah
On Tue, 30 Jan 2007 18:16:03 +0100, Josiah Carlson <jcarlson@uci.edu> wrote:
Chris Rebert <cvrebert@gmail.com> wrote:
Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics...
def append_10(lst=[]): lst.append(10) return lst
I would presumably change it to...
def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst
Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it).
Nowhere has it been proposed to forbid passing None as an argument.
And I never claimed as much. There was a previous post by someone (I can't remember who offered it), saying that replacing 'lst=[]' with 'lst=None' would make it so that a user couldn't signal *something special* with a 'None' argument. I was trying to get across that such reasoning doesn't make sense in this particular context.
The above example code was mine. I was just trying to explain that the proposed semantics should not mistake a caller-provided argument value of None for a no-argument-provided situation and evaluate the default expression. If the proposal were to be implemented naively by literally rewriting the python code in the way that I used to explain the proposal and compiling the rewritten code, this bug could happen. - Jan
Josiah Carlson wrote:
Chris Rebert <cvrebert@gmail.com> wrote:
On 1/28/07, Chris Rebert <cvrebert@gmail.com> wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have. One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where
Collin Winter wrote: this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!), [snip]
Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*.
I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics...
def append_10(lst=[]): lst.append(10) return lst
I would presumably change it to...
def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst
Strings sometimes work nice. Then you have a hint as to what should go there. def append_10(lst='[]'): if lst == '[]': lst = [] lst.append(10) return lst
Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it).
I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use.
- Josiah
I agree, What seems like a gotcha is some situations can be a feature in others. At the most, documenting things like this should be more visible. Like maybe adding a [Gotcha! Mutable arguments...] info box's in the documents. I like those little info box's in manuals, especially if they are written with a bit of humor. Cheers, Ron
participants (10)
-
Adam Olsen -
Chris Rebert -
Collin Winter -
Georg Brandl -
George Sakkis -
Jan Kanis -
Jim Jewett -
Josiah Carlson -
Roman Susi -
Ron Adam