On 1/29/07, Roman Susi wrote:

P.S. However, I may be wrong. In that case my syntax suggestion would be this:

def foo(non_const or []): ...

where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone.

It would also be possible to treat literals (like "[]") as "do it over each time", and more general expressions (like "list()") as they are treated today. Though like Roman, I think this would still be worse than the status quo. -jJ

Adam Olsen wrote:

[snip] Another, more bizarre approach would be to require default arguments be hashable. This would prevent lists from being used, protecting people from mistakes.

If my PEP ends up being rejected, I would fully support this. It wouldn't solve the more general problem, but at least it would prevent a common newbie mistake. - Chris Rebert

Chris Rebert wrote:

...

The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated.

As I see it, the main objection to this is the inversal of the current default semantics: every default argument is treated as if it were mutable, or re-evaluatable more generally. Although mutable default arguments are useful some times and would be nice to have, they are most likely less common than the immutable ones, so the latter should be the default. I'm sure that the python-devs and the BDFL would have thought about it quite a bit when the current semantics were decided, and it's unlikely they'll change their mind now without very good reasons. OTOH, a proposal that leaves the current semantics as is and adds a mechanism to specify default arguments to be evaluated at call-time rather than definition-time would have more chances. Here's a proof-of-concept solution that specifies explicitly the re-evaluatable default expressions as "deferred": import inspect class Deferred(object): def __init__(self, expr): self.expr = expr def eval_deferreds(func): varnames,_,_,defaults = inspect.getargspec(func) num_varnames = len(varnames); num_defaults = len(defaults) def wrapper(*args, **kwds): if len(args) >= num_varnames: # defaults not used here return func(*args,**kwds) f_locals = dict(zip(varnames,args)) used_defaults = min(num_defaults, num_varnames-len(args)) for var,default in zip(varnames[-used_defaults:], defaults[-used_defaults:]): if var in kwds: # passed as keyword argument; don't use the default value = kwds[var] elif not isinstance(default, Deferred): # non re-evaluatable default value = default else: # evaluatable default in f_locals value = eval(default.expr, func.func_globals, f_locals) f_locals[var] = value f_locals.update(kwds) # add any extra keyword arguments return func(**f_locals) return wrapper #======= example ============================== W = 1 # some global @eval_deferreds def f(x, y=Deferred('x**2+W'), z=Deferred('[]')): z.append(x) z.append(y) return z from collections import deque print f(3) # [3,10] W=3; print f(4) # [4,19] print f(4,5) # [4,5] print f(-1, z=deque()) # deque([-1,4]) Regards, George

On 1/29/07, Chris Rebert wrote:

George Sakkis wrote:

...

As I see it, the main objection to this is the inversal of the current default semantics: every default argument is treated as if it were mutable, or re-evaluatable more generally. Although mutable default arguments are useful some times and would be nice to have, they are most likely less common than the immutable ones, so the latter should be the default.

Why? Yes, there _might_ be performance issues (which have yet to be demonstrated or deeply speculated upon), but re-evaluating immutable default arguments wouldn't affect a program's correct operation.

If the underlying intent of your proposal -- that all default arguments be re-evaluated with every call -- were to be approved, there would undoubtedly be a serious performance impact. The alternative is an ugly, narrow-use syntax that seeks to eliminate two lines of boilerplate per default argument, boilerplate that can already be replaced with decorators. As for whether these effects have "yet to be demonstrated", as you say, the burden is on you, the PEP author, to investigate and resolve, mitigate or justify any and all performance changes. Collin Winter

On Tue, 30 Jan 2007 06:40:35 +0100, Chris Rebert wrote:

George Sakkis wrote:

...

I'm sure that the python-devs and the BDFL would have thought about it quite a bit when the current semantics were decided,

...which was probably a while ago. They might reconsider the issue now that some time has passed and they've seen how their decision has worked out. But yes, your analysis is a definite possibility.

Just looked it up, and python has had lexical variables since version 2.1, and default arguments since long before that. (forever?) Without lexical variables it's near impossible to implement re-evaluating default arguments, because the variables those default expressions refer to may no longer be available at function call time. So, The BDFL and other python devs didn't really have a choice but to have the default expressions evaluate at definition time (or implement lexical scopes, which is what has happened by now). Jan

Chris Rebert wrote:

Roman Susi wrote:

...

Hello!

I'd liked to say outright that this bad idea which complicates matters

[skip]

...

P.S. However, I may be wrong. In that case my syntax suggestion would be this:

def foo(non_const or []): ...

where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone. I have not checked, but I believe it is backward compatible. Anyway, could you summarize both contr-argument and this syntax proposal in the PEP?

I don't quite understand exactly how this would work and would like more details on it, but once you've explained it, of course I'd be happy to include it in the next draft.

Simple. def foo(non_const or []): ... is equivalent to def foo(non_const=None): if non_const is None: none_const = [] ... And this will be as before: def foo(non_const=[]): ... Also, I thing that programmers should not use subtle difference between None and other False values, so something like def foo(non_const=None): non_const = none_const or [] is also valid. Another approach (if you want to pursue the feature) could be complication to name binding protocol. a = [] will be as before, but default value assignment could trigger some extra method. So, you can explicitly regulate your instance reaction to default-value assignment: class MyMutableList: ... def __default__(self, old_default): return old_default.copy() Regards, Roman

- Chris Rebert

On Tue, 30 Jan 2007 08:09:37 +0100, Chris Rebert wrote:

I also reworded the sentences using 'shall'. I don't really get why that matters, but what the heck.

Well, it doesn't. Just makes it nicer to read. (As long as the text is still clear, that is.)

Jan Kanis wrote:

...

...

...

...

a = 123 def foo(b=a): a = 2 print a, b foo() 2 123 a = 42 foo() 2 123 In the pep, the a in the default expression would be handled just like any other a in the function body, which means it wil become a _local_ variable. Calling the function would then result in an UnboundLocalError. Just like this in current python: a = 123 def foo(b=None): b = a if b==None else b a = 2 print a foo() Traceback (most recent call last): File "", line 1, in <module> foo() File "", line 2, in foo b = a if b==None else b UnboundLocalError: local variable 'a' referenced before assignment The solution, I think, as I wrote in my previous messages, is to have

Well, I (obviously) like the idea, but your pep misses some important points (and some not so important). The important one is about the scoping of variables in default expressions. The pep says nothing about them. If I read the pep correctly, any variables in default expressions are handled the same as variables in the body of a function. This means the compiler decides if they should be local, lexical or global. If there is an assignment to a variable, the compiler makes it a local, else it finds the right enclosing scope (lexical or global). In the current python, this works fine: the compiler explicitly make variables in default expressions lexical or global variables. This would still break the foo() in my example above, because you can't assign to a lexical variable, or to a global that isn't declared global. Therefore I think the compiler should distinguish between the a in the default expression and the a in the function body, and treat them as two different variables. You can think about it as if the compiler silently renames one of them (without this rename being visible to python code. AFAIK the bytecode is stack based and closure vars are put in some kind of anonymous cell, which means they both don't actually have a name anyway.)

Also, I've been thinking about the part that I suggested about having python distinguish between lexically scoped variables in default expressions and local variables that may have the same name. To clarify my proposal, a demonstration: Current python: a = 4 b = 5 def foo(x = [b]*a): x = copy.deepcopy(x) # one possible workaround to have the same list on every call b = 'bla' # no name clash of this b and b in default arg expr x += [b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] proposed semantics: a = 4 b = 5 def foo(x = [b]*a): # deepcopy() no longer nescessary b = 'bla' # no name clash of this b and b in default arg expr, the compiler distinguishes them x += [b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] This would work the same as this code in the current python. Imagine it as the compiler silently transforming the above code to this code: a = 4 b = 5 def foo(x = _undefined): if x == _undefined: x = [b]*a _b = 'bla' x += [_b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] note: the use of deepcopy assumes that whatever is passed to foo is deepcopyable. With the new semantics this function can be used without these kind of assumptions. However, I've been thinking about weather it is a good idea to have python distinguish between the local and lexical 'b' in the above code, and do the what you can think of as implicit renaming. Making the distinction makes the whole proposal more backward compatible, however I think that if backward compatibility were not an issue, it would be best not to make the distinction and just have the programmer choose a different name for the local variable. This would improve readability as there are no two different variables with the same name in the same piece of code, and decrease language complexity as there is no rule that's best explained in terms of variable name rewriting. Not making the distinction does cause the default arguments proposal to be less backward compatible, but I don't know if that is really a big problem. I assume this pep wil be a py3k pep anyway, so there are going to be incompatibilities anyway. Also, the compiler is perfectly able to see at compile time if a variable used in a default expression and as a local, so it is able to raise a very descriptive error. All in all, I think this part of my proposal is not such a good idea, unless there's going to be some kind of transitional implementation of this pep in py 2.x meta-note: I hope it is clear what I'm trying to say. Basically, I'm clarifying, arguing against, and retracting a part of my own proposal I made in an earlier mail. - Jan

Jim Jewett wrote:

On 1/28/07, Chris Rebert wrote:

...

Naive programmers desiring mutable default arguments often make the mistake of writing the following:

...

def foo(mutable=some_expr_producing_mutable): #rest of function

Yes, it is an ugly gotcha, but so is the alternative.

If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics.

Anyone (ab)using the current semantics with the above construct is or ought to be aware that such usage is unpythonic. Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor. Additionally, I see making the opposite mistake as usually only maybe causing a performance problem, as opposed to causing the program to work incorrectly. As discussed in my proposal, this might be premature optimization. If it is a serious issue (which hasn't been indicated by the discussion so far), we can add syntax for the old/new semantics, which I also mentioned in the proposal.

...

There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables.

I disagree. This is particularly wrong for someone coming from a functional background.

I assume you disagree with the "purpose is much better served by decorators, classes, or local variables" part as opposed to the "default mutable arguments with current semantics have few good uses" part. Please correct me if I'm in error.

A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java.

I'm sorry, but when you have a function sharing state between calls, that just screams to me that you should make it a method of an object so you don't have to store the state in some roundabout way, such as in mutable default arguments. If performance is your concern, the decorator version might perform better (I don't really know), or in the extreme case, you could use a global variable, which is definitely faster. Speed and elegance often come in inverse proportions. Also, I've revised the refactoring in question so that you no longer need to instanciate the class, which at least makes it marginally better.

You see the problem with globals, but decorators are in some sense worse -- a function cannot see its own decorations. At best, it can *assume* that repeating its own name (despite Don't Repeat Yourself) will get another reference to self, but this isn't always true.

I really don't see how the decorator in the PEP is any worse than other decorators in this regard. The problem you describe currently applies to all decorated functions, though functools.wraps might help mitigate this situation.

Programmers used to creating functions outside of toplevel (or class-level) will be more aware of this, and see the suggestion as an indication that python is inherently buggy.

...

def foo(bar=new baz): #code

This would be less bad.

That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.

This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x is None: x=expr' idiom in that it expresses one's intent more clearly. Also, this would at least be a prettier way to spell the idiom even if the reason still needed explaining. Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons. I think I'll add this to the PEP.

...

Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a

This would work if there were any easy way to create a new scope. In Lisp, it makes sense. In python, it would probably be better to just find a way for functions to refer to their own decorations reliably.

This is outside of the scope of my PEP. However, the below improvement should help and you could always use one of the other refactorings to work around this issue.

...

Backwards Compatibility

This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from:

def foo(bar=mutable): #code

to

[a decorator option uses 3 levels of nested functions, which aren't even generic enough for reuse, unless you give up introspection.]

No. It could be done with a (more complex) decorator, if that decorator came with the stdlib (probably in functools)

Agreed, the decorator could be better. I've just enhanced it using functools.wraps, which should help with the introspection issues you raise. Other improvements to the refactoring code in the PEP are welcomed. - Chris Rebert

On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett wrote:

On 1/30/07, Chris Rebert wrote:

...

Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor.

I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process.

Well, it seems that numbers are gonna be needed anyway to convince people. But that's certainly possible. What exactly did you search for, and what did you find? I did some preliminary grepping through the standardlib myself. regex = "def.*\(.*=.*\):", searching through all .py files in /Lib: 2455 matches in total in 474 files out of 998 I manually looked through the first bunch of files, those contained 145 matches. (the first files sorted alphabetically up until and including cookielib.py) By far the most of them had literals as default value. A huge amount of them used the =None idiom, I estimate perhaps one third. 10 matches used variables but will not break under the pep's semantics. I found 1 (one) file that used the current semantics to keep state between calls, in two inner functions. This can easily be changed. So, that leaves 145-12=133 uses of default values that are just constants. If I have time and figure out the right regexes I'll try and come up with some more numbers on the entire stdlib, and the ammount of uses of =None. - Jan

Apologies in advance for the lateness of my reply. I missed a few emails in all the shuffle. Jim Jewett wrote:

On 1/30/07, Chris Rebert wrote:

...

Jim Jewett wrote:

...

On 1/28/07, Chris Rebert wrote:

...

...

...

Naive programmers desiring mutable default arguments often make the mistake of writing the following:

...

...

...

def foo(mutable=some_expr_producing_mutable): #rest of function

...

...

Yes, it is an ugly gotcha, but so is the alternative. If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics.

...

Anyone (ab)using the current semantics with the above construct is or ought to be aware that such usage is unpythonic.

Are you worried about naive programmers or not? [snip]

...

Additionally, I see making the opposite mistake as usually only maybe causing a performance problem, as opposed to causing the program to work incorrectly.

That is true only when the argument is a cache. Generally, resetting the storage will make the program work incorrectly, but the program won't fail on the first data tested -- which means the bug is less likely to be caught.

I am worried about naive programmers. It seems that the opposite mistake (thinking arguments will be evaluated only once, at def-time, rather than at each call) won't have non-performance related effects a good bit of the time. For many of the remaining cases, the person probably is/should be using a constant instead of a default argument. Yes, there will be some cases where badness could happen. However, (1) this is intended for Py3k, so there will already be a bunch of changes for newbies to learn, (2) the chance of weirdness happening is a good bit lower compared to the current state of mutable default arguments.

...

Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor.

I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process.

I have a script to gather statistics on default argument usage in the standard library 90% done. Its results will be included in the next PEP draft.

...

There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or(though less preferred) global variables.

I disagree. This is particularly wrong for someone coming from a functional background.

...

I assume you disagree with the "purpose is much better served by decorators, classes, or local variables" part as opposed to the "default mutable arguments with current semantics have few good uses" part. Please correct me if I'm in error.

I disagree with both. There are many uses for preserving function state between calls. C uses static local variables. Object Oriented languages often use objects.

First, I was referring to using default args for caching being evil, not mutable default arguments in general. Just wanted to clarify in case you misinterpreted me. True, there are many uses for preserving function state between calls, but how much better/clearer is doing so using default arguments as opposed to objects or other approaches? It's an aesthetic decision. IMHO, the other ways seem better. YMMV.

...

...

A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java.

...

I'm sorry, but when you have a function sharing state between calls, that just screams to me that you should make it a method of an object

That's because you drank the OO koolaid. Python tries not to enforce any particular programming style. Its object model happens to be pretty good, but there are still times when OO isn't the answer.

Well, it's a trade-off: support procedural/functional programming a bit more, or make default arguments a bit more intuitive and obsolete the 'foo=None; if foo is None' idiom.

...

If performance is your concern, the decorator version might perform better (I don't really know), or in the extreme case, you could use a global variable, which is definitely faster.

I think there is something wrong with your benchmark.

Sorry, yeah, you're right. I was being a little fast and loose. But the point was that there are (at the price of some elegance) ways to optimize such situations even more.

That said, speed is *not* my concern -- code size is. If I have to add several lines of boilerplate, the code becomes much less readable. That is roughly the same justification you have for not liking the "if arg is None: arg=foo()" idiom. The difference is that you're suggesting adding far more than a line or two.

True, however I'm willing to bet that the boilerplate for evaluating default arguments only once won't be needed nearly as much, as such situations requiring such behavior is not too common. The statistics should help show exactly how much mutable default arguments are exploited in the wild.

...

Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons.

Using a mutable default is almost never for optimization reasons.

Actually, I was more thinking it might be used to stop immutable default arguments from being re-evaluated multiple times, if it turns out that re-evaluations really do have major effects on performance. Now that I think about it, I guess it could be used to get the old behavior for mutable default arguments. - Chris Rebert

Roman Susi schrieb:

Chris Rebert wrote:

...

Jim Jewett wrote:

...

On 1/28/07, Chris Rebert wrote:

/skip/

...

...

...

def foo(bar=new baz): #code

This would be less bad.

That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.

This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x

But if you are concerned with the current None, there could be some other, new False value serving the same need, like:

def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code

or even:

def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code

why not even def foo(x, y, z, bar=..., qux=...): if bar is Ellipsis: # foo ;) Georg

On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson wrote:

Roman Susi wrote:

...

def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code

at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.

The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible.

- Josiah

As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change) - Jan

Josiah Carlson wrote:

"Jan Kanis" wrote:

...

On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson wrote:

...

Roman Susi wrote:

...

def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code

at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.

The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible.

- Josiah

As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change)

Fowards compatible then. That is to say, writing for the new proposed system will break compatibility with older Pythons. On the other hand, using currently-available language syntax and semantics is compatible among the entire range of Pythons available today.

From wikipedia:

"... a product is said to be backward compatible (or downward compatible) when it is able to take the place of an older product, by interoperating with other products that were designed for the older product." "Forward compatibility (sometimes confused with extensibility) is the ability of a system to accept input intended for later versions of itself." So, the right term is that Python 3000 will be in the syntactic aspect we discuss backward compatible because it will accept old syntax too. Regards, Roman

- Josiah

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

!DSPAM:45bff1845411635716952!

"Jan Kanis" wrote:

Forward compatible?? There are exactly zero people on python-dev who care about that. What people care about is that a piece of python code written at the time of 2.4 will still run the same on python 2.5. That's backward compatibility. Nobody cares that a piece of python code using syntax constructs that were new in 2.5 won't run on python 2.4. If you want that, then don't use the new constructs. New syntax constructs get added to the language on just about every major revision, no python major version is forward compatible with a higher major version.

There are many developers who concern themselves with writing code that will work on version x and x+1 of Python. There are also people who like to use "bleeding edge" features while also using an older Python. If there weren't, then 'from __future__ import generators' wouldn't have existed in Python 2.2, 'from __future__ import floor_division' wouldn't exist in Python 2.3+, 'from __future__ import nested_scopes' wouldn't have existed in Python 2.1 or 2.2 (I can't remember), etc. Yes, there is new syntax with each later Python that won't work with previous Pythons, but with the vast majority of syntax changes that have come down (including the with statement in 2.5), the semantics of the new syntax are defined by using the previously existing language. In this case, the 'def foo(x=):' has a translation into current Python, but it turns out that the translation is *exactly what you should be doing today* if you care about correctness.

Python 3.0 is specifically created to allow the backward incompatible changes that people deem nescessary, so neither forward nor backward compatibility matter in that respect. (well, except that it shouldn't become a superhuman task to convert 2.x code to run on python 3.0)

The reason I concerned myself with this in regards to the 'def foo(x=)' proposal is because the ultimate point of this is to make writing code *less* buggy. However, code can be *less* buggy today through the proper application of an if statement or conditional expression. Yes, Py3k does allow for incompatible syntax. But the syntax changes aren't going to be "becase we can", they are going to be "because this way is better". Leaving out an argument isn't better (in the case of the 'def foo(x=):' syntax); it's ambiguous at best, confusing and worthless at worst. We can argue all day about writing compatible code, breaking compatibility, etc., but the fact remains: no developer who has any pull in python-3000 or python-dev has come out in favor of changing default argument semantics in this thread. - Josiah

Chris Rebert wrote:

Roman Susi wrote:

...

Chris Rebert wrote:

...

Jim Jewett wrote:

...

On 1/28/07, Chris Rebert wrote:

/skip/

...

...

...

def foo(bar=new baz): #code

This would be less bad.

That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise.

This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x

But if you are concerned with the current None, there could be some other, new False value serving the same need, like:

def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code

or even:

def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code

at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English.

Those examples are only slightly better than using None. The convention of using None to indicate use of a mutable/non-constant default value is about as clear as your examples.

The point wasn't that None wasn't descriptive/specific enough, but rather why have to write the 'bar=None' and then the 'if bar is None' when 'bar=[]' could be sufficient? Also, this indicates exactly what the default value is, as opposed to None/Missing, which is just an opaque indicator for 'some mutable/non-constant default value'.

Thus, I see no need for 'Missing' or a similar constant.

And I think it is overkill to add new syntax there. For example, if the function or method is near the external (human or network) interface border it is good to check ALL arguments anyway. So the extra if baz is Missing: baz = [] doesn't spoil the impression: def foo(x, y, z, baz=, qux=): ... if baz is Missing: baz = [] ... if type(baz) != type([]) or len(baz) > 1000 or min(baz) < 0 or ...: raise ... #code What I'd wanted to say is that =None or =Missing is not just "indicator for 'some mutable/non-constant default value'." but may also mean that the argument is subject to further constraints and thus funtion code has checks for that. So, instead of having clear indicator of missing argument we have some fresh instance of a list or dict. Also, in many situation another approach is used nowadays: def foo(x, y, z, **args): ... baz = args.get("baz", []) qux = args.get("qux", {}) Which has the effect proto-PEP author would like to achieve. Of course, no automatic tools are going to make type checks with that helpful function signature. Also, while its offtopic in this post, I'd really liked Python 3000 to have more unification for dict vs attribute access because there are use cases (Storage in web.py, Namespace in mxTools - I am sure many other projects have similar things) which single purpose is to have both dict and attribute syntax in one data object interchangeably: def foo(**args): ... baz = args.baz qux = args["qux"] Regards, Roman

- Chris Rebert

Collin Winter wrote:

On 1/28/07, Chris Rebert wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.

One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!), and doesn't allow you to refer to other arguments in determining the default value (think 'self' as a hint for how this could be useful). Also, I just think it's plain nicer/prettier if decorators don't have to be used for this.

Also, you haven't talked at all about how all this might be accomplished. You say

...

Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time.

but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments?

The "Reference Implementation" section of the PEP discusses this. I don't personally know Python's internals, but I imagine this proposal would just change how default arguments are compiled and might entail some changes to the interpreter's argument-processing machinery. As Jan Kanis pointed out, the PEP does need to elaborate on exactly what kind of variables the default arguments use/are. I'm working on revising this in the next draft. - Chris Rebert

Collin Winter wrote:

On 1/29/07, Chris Rebert wrote:

...

Collin Winter wrote:

...

On 1/28/07, Chris Rebert wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.

One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work.

Tomer wasn't proposing that any of those decorators be included in the standard library. Those cases where one of decorators isn't sufficient as written? Modify it, use it in your code, post it to the cookbook. Not every n-line function should be turned into syntax.

Right, but for something used so frequently, should you really have to look through N decorator recipes, figure out which one you want, and then modify it, as opposed to a change in semantics that obviates the need for such decorators altogether?

...

His second decorator involves using lambdas (ick!), and doesn't allow you to refer to other arguments in determining the default value (think 'self' as a hint for how this could be useful).

You can't do that as things are, so that's not a strike against the decorator-based approach.

Valid point. It's actually more pro-reevaluation than it is anti-decorator.

...

...

but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments?

The "Reference Implementation" section of the PEP discusses this. I don't personally know Python's internals, but I imagine this proposal would just change how default arguments are compiled and might entail some changes to the interpreter's argument-processing machinery.

As you work on this, think about how you'll explain the new semantics in the documentation. Rewriting http://docs.python.org/ref/calls.html and http://docs.python.org/ref/function.html -- paying particular attention to paragraphs 2-5 in the former -- would be a good start.

Sweet! Thanks for the link. - Chris Rebert

Chris Rebert wrote:

Collin Winter wrote:

...

On 1/28/07, Chris Rebert wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have.

One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!),

[snip] Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*. I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics... def append_10(lst=[]): lst.append(10) return lst I would presumably change it to... def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it). I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use. - Josiah

Chris Rebert wrote:

Josiah Carlson wrote:

...

Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*.

No, it's just that why use several different decorators when a slight change in semantics renders them all obsolete? Why have to use a decorator/lambda/conditional and remember when to use which? I'm trying to generalize the pattern of "reevaluate/copy default values each time they're required" into new behavior for default arguments. Indeed, as I stated in another email (no idea which one, there's been too many), I wholeheartedly support adding some of those decorators to the standard library should my PEP get rejected.

The conditional works in all cases. The lambda works in all cases. The set of decorators seems to need to be tuned depending on the situation. Me, I would just use a conditional or even a plain if statement, as it is already sufficiently universal to handle every case. Discussions about 'but then I can't mask global names with local names' are going to fall on deaf ears; it's seen as poor form, and discussing what one can do using poor Python form is not topical.

...

I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics...

def append_10(lst=[]): lst.append(10) return lst

I would presumably change it to...

def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst

Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it).

Nowhere has it been proposed to forbid passing None as an argument.

And I never claimed as much. There was a previous post by someone (I can't remember who offered it), saying that replacing 'lst=[]' with 'lst=None' would make it so that a user couldn't signal *something special* with a 'None' argument. I was trying to get across that such reasoning doesn't make sense in this particular context.

The basic core of the proposal is to have Python compile the code such that the boilerplate is either removed entirely (if the new semantics are the default) by no longer being necessary, or dramatically shortened via the addition of a new keyword (which would indicate the new semantics). Yes, you can manually transform your code to use the idiom you mention. But why should you have to? Why not have Python do the heavy-lifting for you?

Heavy lifting? One line change and one line addition is heavy lifting? There has got to be an episode of Monty Python embedded in this thread somewhere, because I can't help laughing at your claim.

...

I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use.

I'm not here to argue that people shouldn't be made to understand the difference between mutable and immutable values. They definitely should know the difference. I'm just advocating a language change to make certain kinds of functions less verbose.

My point is that the change results in *certain* functions being trivially less verbose. I've spent more time working around re's call semantics than I have dealing with (incorrect assumptions about Python's) default argument semantics. As I don't see a need to change re, I don't see a need to change default argument semantics. - Josiah

Josiah Carlson wrote:

Chris Rebert wrote:

...

...

On 1/28/07, Chris Rebert wrote: Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have. One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where

Collin Winter wrote: this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!), [snip]

Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*.

I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics...

def append_10(lst=[]): lst.append(10) return lst

I would presumably change it to...

def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst

Strings sometimes work nice. Then you have a hint as to what should go there. def append_10(lst='[]'): if lst == '[]': lst = [] lst.append(10) return lst

Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it).

I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use.

- Josiah

I agree, What seems like a gotcha is some situations can be a feature in others. At the most, documenting things like this should be more visible. Like maybe adding a [Gotcha! Mutable arguments...] info box's in the documents. I like those little info box's in manuals, especially if they are written with a bit of humor. Cheers, Ron