func attribute & default arg

Hello, I think there is some confusion on the topic of static variables in relation to default arguments and function attributes. Or, maybe, my point of view is different from the one of the majority on python-ideas. (a rather long post) To sum up, I think the core of the issue is not about static/dymamic evaluation, or compile time/run time; rather it lies in the fact that python internally does not separate a default *value* from the corresponding local *variable*. What may be, for the programmer, the meaning and purpose of a default argument? I think there is nothing misleading in the fact that it is evaluated at definition time. Instead, this is precisely the way we see them, otherwise there would be no gotcha and we would not discuss. This semantics is properly denoted, I guess, by a default beeing part of the function definition's headline: def f(arg, default=value): ....... Once set, a default becomes a kind of constant attribute of the function. So that the gotcha happens precisely when this expectation is deceived because the local variable called 'default' may change back the previously defined value. To address this issue, having a default value recomputed on every call is not only overkill, but also semantically aside the point. What we need is only *caching* a default value while the function def is first evaluated. A possible implementation (probably requiring few changes) may be to add a kind of __defaults__ dict attribute to functions. This would be filled at definition time and used at call time. Even if probably not needed, for python provides better ways, having it as an available attribute also lets programmers tweak a func's guts if they like it -- e.g. for runtime updatable defaults: func.__defaults__["default_arg"] = newDefaultValue() With this semantics, we are free to use and update local variables safely, without worrying of changing the default value. Here is a short and probably non-exhaustive review of common uses of defaults. -1- real default arg which value is immutable No issue. -2- real default arg which value is mutable but not updated in func body No issue. With the above proposal, we do not need to worry anyway; and we can change the code later safely to case -3-. I find this important. -3- real default arg which value is mutable and updated in func body Present implementation: gotcha. Proposal: default argument and local var point to distinct values. No more issue. Possibly use a __defaults__ dict attribute. -4- default used as static var A common use of mutable defaults is a workaround for python seemingly not having static variables. Actually, python programmers have them in the form of function attributes. Since this feature seems +/- unknown (undocumented?), some programmers write for instance: def f(arg, l=[]): <update l> while the following better expresses the semantics, imho, because l is not a real argument: def f(arg): <update f.l> f.l = [] Generators can also be used in cases where the purpose of the func is (mainly) to return successive values. def f(arg): l = [] while True: <update l> yield something -5- func factory In the case of a factory, default arguments are used as function definition *parameters*. Similarly to the previous case, they are rather function attributes, with the difference that this time they are constant ones. Anyway, the proposal does not change the semantics here because each generated function has its own parameter. Still, a common idiom using a dummy default, such as: def paramPower(exponent): def power(number, exponent=exponent): return number ** exponent return power would be better expressed, without any trick: def paramPower(exponent): def power(number): return number ** power.exponent power.exponent = exponent return power -6- variable defaults If we want a runtime evaluation of a default argument, we usually express it in the function body and use a sentinel. This would not change, and anyway it seems to make the intent clearer than in the case where the default would be silently recomputed (read: "explicit better than implicit"): from random import random # provides runtime default number_count = 3 # may also be variable UNDEF = object() def f(i, numbers=UNDEF): # case numbers not provided: must be evaluated at runtime if numbers is UNDEF: numbers = [int(random()*10) for i in range(number_count)] # change pointed number numbers[i] = int(random()*10) print numbers numbers = [1,2,3] f(0, numbers) ; f(2, numbers) ; f(0) ; f(2) ==> [8, 2, 3] [8, 2, 2] [5, 3, 5] [8, 4, 1] Using the proposal of a default attribute, an alternative would be to re-compute the value in calling code -- only when needed: def f(i, numbers=[random() for i in range(number_count)]): # change pointed number numbers[i] = random() print numbers ....... if cond: f.__defaults__['numbers'] = [random() for i in range(number_count)] f(i) A transition phase could be planned toward cached defaults, possibly with proper warnings: "Warning: use of a mutable default value updated in the function body. From python 3.x, default values will be cached so as not to change between calls. If your intention is instead to have a kind of static variable, rather set an attribute on the function." Now, why are func attributes rarely used; while, as shown above, they provide a clear and handy way to express several semantic patterns -- and defaults are (mis)used instead? I wonder whether we could make them more obvious using a dedicated syntax. In the thread about default arguments, a proposed syntax, to have defaults dynamically reevaluated at each call (which I find *wrong*), was to reuse the keyword "yield". The meaning of this idiom is rather for me the creation of some data at definition time. As a consequence, I found it a rather good possibility to denote the creation of a func attribute. : def f(arglist) yield attr=val: for more attributes, reuse import-like syntax: def f(arglist) yield attr1=val1, attr2=val2...: Having them defined outside the parens properly differenciate arguments from attributes (and the content of parens is already complicated enough with defaults and '*' and '**' thingies). Also, the "yield keyword not only suggests the creation of data, but also is used in contexts (generators) where state is kept between function calls. For instance, the following code using a func with a custom attribute: startup_numbers = [4, 5] def f(number, sorting=False): f.numbers.append(number) if sorting: f.numbers.sort() print f.numbers f.numbers = startup_numbers could be expressed with a func def header as follows: def f(number, sorting=False) yield numbers=startup_numbers: without the need to separately set the attribute on the func. Sure, we do not need that at all. Moreover, when the func happens to be a method instance, then the attribute better fits on the instance object. I'm just looking for a way to let programmers be (more) aware of func attributes. To have cached default solving the common gotcha, we need python programmers to use defaults only as defaults, and use other solutions for semantics such as static vars. So that the transition be easier. Denis ------ la vita e estrany

spir wrote:
Good try, but you seem to have missed the basic point that underlies at several ways that newbies get tripped up*. Python is an *object* based language, not a *value* based language. By 'value', I mean the information carried by the object. Objects have an identity, values do not. This is a defining characteristic of Python. If one wants an immutable value language, Python is not the right choice. For immutable objects, the difference is nearly invisible. For mutable objects, which generally do not exist in mathematics (which is generally timeless) the difference is crucial to understand. When people are tripped up by their 'intuition', it is sometimes intuition based on math that does not apply to mutable objects. Names are bound to objects, not to values. For functions, parameters are local names, arguments are objects (and not values). Functions may have default objects, not 'default values' (as you say above and elsewhere). If a default object is mutable, it has an initial value, but not a constant value. Function objects already have an attribute that is a tuple of default objects. (Parameter names, without and with defaults, are part of the code object.) Python 3.0.1 ,,, on win32
def f(a, b=1, c='1'): pass
f.__defaults__ (1, '1')
Terry Jan Reedy * Mutable objects with multiple names, mutable objects passed as argument, mutables used as defaults, constructing initialized lists of lists.

Le Thu, 14 May 2009 17:33:36 -0400, Terry Reedy <tjreedy@udel.edu> s'exprima ainsi:
Thank you for this clear reply. Actually, I 101% agree with the object/value discrimination (and even think there should be both in a prog. language, based on the fact that objects have an id and consequently are referenced.)
Right. But this does not imply that a local variable must point to the same object that deals as default. This is actually the point I tried to make; but obviously not clear enough. It simply requires that while evaluating a function definition python caches defaults in a separate "storage place" that will not be later affected by the use of local variables. E.g. updating a 'number_list' local name binding should not update a default value/object for the corresponding parameter, if any.
Didn't know about that, but it makes a change even easier. (Unfortunately, I haven't py3 installed to play with this.) So, let's change the example to: def storeNums(num, numList=[1,2,3]): numList.append(num) print "list of nums: %s" % numList The issue is obviously that, at call time, python will bind the func local name representing the *actual* parameter numList to the default object (*) defined together with and for the *formal* parameter of the same name numList. So that not only it may be changed, but this changed object will stay as default object and propagate to later calls. The evident solution is to let python get the "data" (to avoid "value") instead of refering to the object -- which implies copying. End of issue? Denis (*) I use your term and agree it's better when talking of python, but actually it seems that most say "default value", no? Also, value is widely used in the python community, first in official docs, i guess.
Terry Jan Reedy

spir wrote:
There are only two ways to make that happen though: 1. Evaluate the default argument expression at function call time instead of definition time 2. Actually *copy* the default argument to create the object referenced by the parameter on a given invocation Since option 1 will work for any arbitrary object, while option 2 is limited to objects which support copy.copy (or copy.deepcopy, depending on the copying semantics chosen), the "late evaluation of default arguments" (option 1) approach is the more promising angle of attack by far. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Thu, 14 May 2009 09:23:39 pm spir wrote:
You mean like this?
This has existed since at least Python 1.5. [...]
This makes no sense. Python already caches defaults. If you modify a cached object, you will see the modifications in the next call. You have your understanding completely backwards: what some people want is for Python to *stop* caching default values. Denis, with respect, I think your idea is too raw for this list. I don't wish to discourage you, but should try posting to the general python mailing list first, for feedback and suggestions. I think this confusion about cached objects is the sort of fundamental misunderstanding that is best discussed on the general python mailing list rather than here. -- Steven D'Aprano

spir wrote:
Good try, but you seem to have missed the basic point that underlies at several ways that newbies get tripped up*. Python is an *object* based language, not a *value* based language. By 'value', I mean the information carried by the object. Objects have an identity, values do not. This is a defining characteristic of Python. If one wants an immutable value language, Python is not the right choice. For immutable objects, the difference is nearly invisible. For mutable objects, which generally do not exist in mathematics (which is generally timeless) the difference is crucial to understand. When people are tripped up by their 'intuition', it is sometimes intuition based on math that does not apply to mutable objects. Names are bound to objects, not to values. For functions, parameters are local names, arguments are objects (and not values). Functions may have default objects, not 'default values' (as you say above and elsewhere). If a default object is mutable, it has an initial value, but not a constant value. Function objects already have an attribute that is a tuple of default objects. (Parameter names, without and with defaults, are part of the code object.) Python 3.0.1 ,,, on win32
def f(a, b=1, c='1'): pass
f.__defaults__ (1, '1')
Terry Jan Reedy * Mutable objects with multiple names, mutable objects passed as argument, mutables used as defaults, constructing initialized lists of lists.

Le Thu, 14 May 2009 17:33:36 -0400, Terry Reedy <tjreedy@udel.edu> s'exprima ainsi:
Thank you for this clear reply. Actually, I 101% agree with the object/value discrimination (and even think there should be both in a prog. language, based on the fact that objects have an id and consequently are referenced.)
Right. But this does not imply that a local variable must point to the same object that deals as default. This is actually the point I tried to make; but obviously not clear enough. It simply requires that while evaluating a function definition python caches defaults in a separate "storage place" that will not be later affected by the use of local variables. E.g. updating a 'number_list' local name binding should not update a default value/object for the corresponding parameter, if any.
Didn't know about that, but it makes a change even easier. (Unfortunately, I haven't py3 installed to play with this.) So, let's change the example to: def storeNums(num, numList=[1,2,3]): numList.append(num) print "list of nums: %s" % numList The issue is obviously that, at call time, python will bind the func local name representing the *actual* parameter numList to the default object (*) defined together with and for the *formal* parameter of the same name numList. So that not only it may be changed, but this changed object will stay as default object and propagate to later calls. The evident solution is to let python get the "data" (to avoid "value") instead of refering to the object -- which implies copying. End of issue? Denis (*) I use your term and agree it's better when talking of python, but actually it seems that most say "default value", no? Also, value is widely used in the python community, first in official docs, i guess.
Terry Jan Reedy

spir wrote:
There are only two ways to make that happen though: 1. Evaluate the default argument expression at function call time instead of definition time 2. Actually *copy* the default argument to create the object referenced by the parameter on a given invocation Since option 1 will work for any arbitrary object, while option 2 is limited to objects which support copy.copy (or copy.deepcopy, depending on the copying semantics chosen), the "late evaluation of default arguments" (option 1) approach is the more promising angle of attack by far. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Thu, 14 May 2009 09:23:39 pm spir wrote:
You mean like this?
This has existed since at least Python 1.5. [...]
This makes no sense. Python already caches defaults. If you modify a cached object, you will see the modifications in the next call. You have your understanding completely backwards: what some people want is for Python to *stop* caching default values. Denis, with respect, I think your idea is too raw for this list. I don't wish to discourage you, but should try posting to the general python mailing list first, for feedback and suggestions. I think this confusion about cached objects is the sort of fundamental misunderstanding that is best discussed on the general python mailing list rather than here. -- Steven D'Aprano
participants (5)
-
Georg Brandl
-
Nick Coghlan
-
spir
-
Steven D'Aprano
-
Terry Reedy